2 Commits

Author	SHA1	Message	Date
Phil Wang	a851168633	make youtokentome optional package, due to reported installation difficulties	2022-06-01 09:25:35 -07:00
Phil Wang	7cf1637d24	bring in the simple tokenizer released by openai, but also plan on leaving room for custom tokenizer with yttm	2022-04-12 09:23:17 -07:00