-
Notifications
You must be signed in to change notification settings - Fork 467
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial commit for CheckpointManager #5678
Conversation
The functionality here doesn't depend on it. For distributed checkpointing, the main use case of that PR is to discover the master IP for the CPU process group. The stronger use case is going to be distributed data loading. I'll sync with Yeounoh if we're OK to land it now, otherwise I can convert to draft until it's needed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Please make sure this is up to date with the design doc.
5a48879
to
46f9706
Compare
async_queue_size: The size of the execution queue which processes async | ||
checkpoints. This should be a small value to ensure training doesn't | ||
get too far ahead of the last finished checkpoint, but increasing | ||
the value to 2 can unblock training when there are transient | ||
network issues which slow down the active checkpoint. | ||
Default: 1, which only allows a single async checkpoint to be | ||
pending at a time. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc @alanwaketan
* Initial commit for CheckpointManager * Update documentation for async * Fix typo
* Initial commit for CheckpointManager * Update documentation for async * Fix typo
* Initial commit for CheckpointManager * Update documentation for async * Fix typo
* Initial commit for CheckpointManager * Update documentation for async * Fix typo
* Initial commit for CheckpointManager * Update documentation for async * Fix typo
* Initial commit for CheckpointManager * Update documentation for async * Fix typo
* Initial commit for CheckpointManager * Update documentation for async * Fix typo
This adds the boilerplate code and documentation for CheckpointManager, a higher-level wrapper which interfaces with the
torch.distributed.checkpoint
APIs. CheckpointManager will provide some key functionality: