Migrate Tokenizer Components to utilize pytorch-labs/tokenizers #1440
Labels
actionable
Items in the backlog waiting for an appropriate impl/fix
enhancement
New feature or request
good first issue
Good for newcomers
triaged
This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Comments
Jack-Khuu
added
enhancement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
🚀 The feature, motivation and pitch
@larryliu0820 has created a new shared repository for hosting tokenizer definitions.
The initial migration attempt was reverted in #1414 due to a tokenizer issue flagged in #1413, but should be straightforward to debug and reland
Task: Taking inspiration from #1401, reattempt this migration
Alternatives
No response
Additional context
No response
RFC (Optional)
No response
The text was updated successfully, but these errors were encountered: