Releases: tiktoken-go/tokenizer
Releases · tiktoken-go/tokenizer
Support for O1 models
Merge pull request #10 from ando-masaki/o1-preview-and-mini-fix Added o1-preview and o1-mini
Added support for GPT-4o and o200k base models
This release adds support for the latest models from OpenAI
Lazy loading maps
Vocabulary maps are now lazy loaded when use for the first time. This should minimize the amount of memory consumed and start-up time.
v0.0.8: 🪄 clean up the bpe algorithm
Renamed variables struct to reflect what they do a little better.