refactor: use tiktoken for token calculation#12416

Merged

laipz8200 merged 2 commits intomainfrom refactor/back-to-calculate-tokens-directly-and-use-tiktoken

Jan 7, 2025

+15-12

Commits on Jan 7, 2025

refactor: replace GPT2Tokenizer implementation with tiktoken for improved performance
laipz8200
committed
fix: remove unnecessary verbose flag from GPT2Tokenizer encode method
laipz8200
committed