New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use AI21 local tokenizer when available #1772

Closed

yifanmai opened this issue Aug 4, 2023 · 1 comment · Fixed by #2775

Labels

good first issue models p2

Collaborator

yifanmai commented Aug 4, 2023

This would significantly speed up evaluation.

yifanmai mentioned this issue

access to Tokenizer on SageMaker-deployed models AI21Labs/SageMaker#7

Closed

yifanmai added good first issue p2 models labels

Collaborator Author

yifanmai commented Feb 7, 2024

The AI21 tokenizer is available here: https://github.com/AI21Labs/ai21-tokenizer

yifanmai mentioned this issue

Add local tokenizer for AI21 #2775

Merged

yifanmai closed this as completed in #2775

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment