Add a config not to shuffle merged dataset #1394
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Resolves #1393 |
| if cfg.not_shuffle_merged_datasets: | ||
| LOG.info("NOT shuffling merged pretraining datasets") | ||
| else: | ||
| dataset = dataset.shuffle(seed=seed, buffer_size=buffer_size) |
There was a problem hiding this comment.
I am unsure if this is intended to shuffle the pre-training dataset (it is a single dataset) within the buffer size?
winglian
previously requested changes
Mar 12, 2024
Co-authored-by: Wing Lian <wing.lian@gmail.com>
|
I am a bit curious. This only disables shuffling when merging datasets. It is still intended to shuffle within dataset? |
|
Does this PR require some more changes to be merged? PTAL |
NanoCode012
approved these changes
Mar 19, 2024
seungduk-yanolja
added a commit
to Y-IAB/axolotl
that referenced
this pull request
Mar 19, 2024
…kip ci] * Add a config not to shuffle merged dataset * Update README.md * Update src/axolotl/utils/config/models/input/v0_4_1/__init__.py Co-authored-by: Wing Lian <wing.lian@gmail.com> * invert the condition name * update README * info -> debug --------- Co-authored-by: Wing Lian <wing.lian@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a config not to shuffle merged dataset
Description
Added a config named
not_shuffle_merged_datasets, which I have been using in my fork for a long time :)Motivation and Context
When training a model to expand its vocab with non-English tokens, I usually start with parallel corpora and then train it on web-crawled or something suitable for pre-training.
It is better giving the user to have an option not to shuffle the merged datasets anyway.
How has this been tested?
This config has been used in my fork for a long time and I verified that it works by seeing the loss graph.
Screenshots (if appropriate)
N/A
Types of changes
New feature (non-breaking change which adds functionality)