Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a method to query the remaining tokens of key #380

Closed
gouguoyin opened this issue Jun 16, 2023 · 4 comments
Closed

Add a method to query the remaining tokens of key #380

gouguoyin opened this issue Jun 16, 2023 · 4 comments
Labels
duplicate This issue or pull request already exists enhancement New feature or request

Comments

@gouguoyin
Copy link

Add a method to query the total and remaining tokens of key
@anhao
Copy link

anhao commented Jun 16, 2023

https://github.com/pkoukk/tiktoken-go

@vvatanabe
Copy link
Collaborator

FYI

If it would be possible to bring tokenization with zero (or minimal) dependencies — I'm all for merging it. Otherwise, I think it makes sense to implement it in a separate repo.

#62 (comment)

@kamushadenes
Copy link

kamushadenes commented Jun 18, 2023 •
edited
Loading

I used https://github.com/j178/tiktoken-go (Rust bindings), replaced with pure-Go https://github.com/pkoukk/tiktoken-go with no issues at all.

@vvatanabe vvatanabe added enhancement New feature or request duplicate This issue or pull request already exists labels Jun 30, 2023
@vvatanabe
Copy link
Collaborator

If there's more to discuss, please post it in issue #62.

@vvatanabe vvatanabe mentioned this issue Jul 3, 2023
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
duplicate This issue or pull request already exists enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kamushadenes @anhao @vvatanabe @gouguoyin