Skip to content

Releases: tiktoken-go/tokenizer

Support for O1 models

22 Nov 22:16
2ad5811
Compare
Choose a tag to compare
Merge pull request #10 from ando-masaki/o1-preview-and-mini-fix

Added o1-preview and o1-mini

Added support for GPT-4o and o200k base models

13 Sep 15:32
Compare
Choose a tag to compare

This release adds support for the latest models from OpenAI

Lazy loading maps

10 May 19:49
7f65fe7
Compare
Choose a tag to compare

Vocabulary maps are now lazy loaded when use for the first time. This should minimize the amount of memory consumed and start-up time.

v0.0.8: 🪄 clean up the bpe algorithm

09 Apr 21:48
Compare
Choose a tag to compare
Renamed variables struct to reflect what they do a little better.