Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Condition the meta initialization for hf_causal_lm on pretrain #1397

Merged
merged 1 commit into from
Jul 26, 2024

Conversation

irenedea
Copy link
Contributor

@irenedea irenedea commented Jul 26, 2024

Before this change, there is an error if the pretrained_model_name_or_path does not have model weights (like safetensors, etc.)

After this change, we can pass in a pretrained_model_name_or_path that only has config files if you are not initializing the model with pretrained weights.

Manually tested in interactive via:

huggingface-cli download meta-llama/Meta-Llama-3-8B config.json --local-dir /local-model/

and setting pretrained_model_name_or_path to /local-model

@irenedea irenedea requested a review from a team as a code owner July 26, 2024 13:43
@irenedea irenedea enabled auto-merge (squash) July 26, 2024 16:50
@irenedea irenedea merged commit bb385f6 into mosaicml:main Jul 26, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants