Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix(tokenizing): Use multi-core #293

Merged
merged 1 commit into from
Jul 19, 2023

Conversation

NanoCode012
Copy link
Collaborator

@NanoCode012 NanoCode012 commented Jul 19, 2023

This improves speed for tokenizer and adds a progress bar.

Benchmark:

dataset old new
teknium/GPT4-LLM-Cleaned ~70s < 10s
WizardLM/WizardLM_evol_instruct_V2_196k ~5 min ~55s

Note: just internally counting, nothing strict.

Future PR for more speed can:

  • use batching. (requires changing class to not use iter)
  • fix Dataset.from_list and shuffle (do we need to save to a list? why not save to a dataset directly to cut off this part)
  • ConstantIterList can be improved

Full credit to: neverendingtoast

Copy link
Collaborator

@winglian winglian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥

@NanoCode012 NanoCode012 merged commit 28fd429 into axolotl-ai-cloud:main Jul 19, 2023
@NanoCode012 NanoCode012 deleted the fix/tokenize-speed branch July 19, 2023 02:02
@NanoCode012
Copy link
Collaborator Author

One note: this removes the error if dataset is empty. We can simply just add into it the data.py file instead.

@winglian winglian added the enhancement New feature or request label Jul 22, 2023
mkeoliya pushed a commit to mkeoliya/axolotl that referenced this pull request Dec 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants