Changed convert-pt-to-ggml.py to use .tiktoken tokenizer files#725

Merged

ggerganov merged 1 commit intoggerganov:masterfrom ivan-gorin:master

Apr 14, 2023

+7-11

Commits on Apr 6, 2023

Changed convert-pt-to-ggml to use .tiktoken tokenizer files
ivan-gorin
committed