Update tiktoken-rs to 0.6 and refactor token encoding
#60
| Job | Run time |
|---|---|
| 21s | |
| 19s | |
| 40s |
tiktoken-rs to 0.6 and refactor token encoding
#60
| Job | Run time |
|---|---|
| 21s | |
| 19s | |
| 40s |