Code

Add support: o200k_base tokenizer.#37

Code

Merged

IAPark merged 1 commit intoIAPark:mainIAPark/tiktoken_ruby:mainfrom CaffeeLake:o200kCaffeeLake/tiktoken_ruby:o200kCopy head branch name to clipboardMay 16, 2024

+121-128

Commits

Commits on May 16, 2024

• Add support: o200k_base tokenizer.

  CaffeeLake

  committed
  95b1fa9

  Copy full SHA for 95b1fa9