bring in the simple tokenizer released by openai, but also plan on leaving room for custom tokenizer with yttm

Browse Source

...

This commit is contained in:

Phil Wang

2022-04-12 09:23:17 -07:00

parent 4ff6d021c9

commit 7cf1637d24

5 changed files with 262393 additions and 10 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

262145

dalle2_pytorch/data/bpe_simple_vocab_16e6.txt Normal file

File diff suppressed because it is too large Load Diff