Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: ChatGPT can't load the optimizer weights saved before on Multi-GPU Zero2 mode #3250

Closed
hijkzzz opened this issue Mar 26, 2023 · 1 comment
Closed

[BUG]: ChatGPT can't load the optimizer weights saved before on Multi-GPU Zero2 mode #3250

hijkzzz opened this issue Mar 26, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@hijkzzz
Copy link

hijkzzz commented Mar 26, 2023

🐛 Describe the bug

The root cause is that the state_dict does not match.
We should gather the optimizer weights before save funciton.

Environment

No response
@hijkzzz hijkzzz added the bug Something isn't working label Mar 26, 2023
@binmakeswell
Copy link
Member

@hijkzzz Thanks for your proposal, we will update save/load next week.

@hijkzzz hijkzzz closed this as completed Apr 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hijkzzz @binmakeswell