-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
zero3 performance optimizations #3622
Merged
Merged
Commits on May 22, 2023
-
Configuration menu - View commit details
-
Copy full SHA for e1e4b75 - Browse repository at this point
Copy the full SHA e1e4b75View commit details -
Prevent evaluation of debug strings
Debug strings are evaluated even when logging is disabled
Configuration menu - View commit details
-
Copy full SHA for e3dbb7a - Browse repository at this point
Copy the full SHA e3dbb7aView commit details
Commits on May 29, 2023
-
Use contiguous gradients tensor reduce scatter between ranks
Use allreduce instead of reduce scatter. lower cpu overhead.
Configuration menu - View commit details
-
Copy full SHA for bd4d724 - Browse repository at this point
Copy the full SHA bd4d724View commit details
Commits on May 30, 2023
-
move overflow tracker to optimizer.step
Don't check overflow in gradients for every bucket. Do overflow chack once on grad flat buffer just before optimizer step
Configuration menu - View commit details
-
Copy full SHA for 9cf826d - Browse repository at this point
Copy the full SHA 9cf826dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6847b30 - Browse repository at this point
Copy the full SHA 6847b30View commit details
Commits on Jun 1, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 9b80d60 - Browse repository at this point
Copy the full SHA 9b80d60View commit details
Commits on Jun 2, 2023
-
Configuration menu - View commit details
-
Copy full SHA for a813c77 - Browse repository at this point
Copy the full SHA a813c77View commit details -
Configuration menu - View commit details
-
Copy full SHA for b0bfdc1 - Browse repository at this point
Copy the full SHA b0bfdc1View commit details
Commits on Jun 5, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 7b83224 - Browse repository at this point
Copy the full SHA 7b83224View commit details
Commits on Jun 6, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 89add18 - Browse repository at this point
Copy the full SHA 89add18View commit details -
Configuration menu - View commit details
-
Copy full SHA for 16e6f1c - Browse repository at this point
Copy the full SHA 16e6f1cView commit details
Commits on Jun 7, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 290bdbd - Browse repository at this point
Copy the full SHA 290bdbdView commit details
Commits on Jun 8, 2023
-
Configuration menu - View commit details
-
Copy full SHA for df88b94 - Browse repository at this point
Copy the full SHA df88b94View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.