-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Yuya/add checkpoints section #9329
Conversation
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
* Add 'Quantized Checkpoints' section Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Address review comments Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com>
* Describe shardings and entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Strategies, optimizers, finalize entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Transformations Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Integration Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add link from intro Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply grammar suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Explain the example Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply review suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add zarr and torch_dist explanation --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Co-authored-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, have suggested some small changes.
Extra note:
In user_guide.rst
, maybe prepend all script paths with NeMo/scripts/checkpoint_converters ?
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM now, thank you
* Add checkpoints section Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix title Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Add section on ".qnemo" checkpoints (#9503) * Add 'Quantized Checkpoints' section Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Address review comments Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Distributed checkpointing user guide (#9494) * Describe shardings and entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Strategies, optimizers, finalize entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Transformations Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Integration Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add link from intro Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply grammar suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Explain the example Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply review suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add zarr and torch_dist explanation --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * add subsection Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Update docs/source/checkpoints/intro.rst Co-authored-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix code block Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * formatting Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: Jan Lasek <janek.lasek@gmail.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com>
* Add checkpoints section Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix title Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Add section on ".qnemo" checkpoints (NVIDIA#9503) * Add 'Quantized Checkpoints' section Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Address review comments Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Distributed checkpointing user guide (NVIDIA#9494) * Describe shardings and entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Strategies, optimizers, finalize entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Transformations Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Integration Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add link from intro Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply grammar suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Explain the example Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply review suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add zarr and torch_dist explanation --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * add subsection Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Update docs/source/checkpoints/intro.rst Co-authored-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix code block Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * formatting Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: Jan Lasek <janek.lasek@gmail.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Co-authored-by: Chen Cui <chcui@nvidia.com>
* Add checkpoints section Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix title Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Add section on ".qnemo" checkpoints (#9503) * Add 'Quantized Checkpoints' section Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Address review comments Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Distributed checkpointing user guide (#9494) * Describe shardings and entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Strategies, optimizers, finalize entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Transformations Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Integration Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add link from intro Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply grammar suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Explain the example Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply review suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add zarr and torch_dist explanation --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * add subsection Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Update docs/source/checkpoints/intro.rst Co-authored-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix code block Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * formatting Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: Jan Lasek <janek.lasek@gmail.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
* Add checkpoints section Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix title Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Add section on ".qnemo" checkpoints (NVIDIA#9503) * Add 'Quantized Checkpoints' section Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Address review comments Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Distributed checkpointing user guide (NVIDIA#9494) * Describe shardings and entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Strategies, optimizers, finalize entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Transformations Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Integration Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add link from intro Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply grammar suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Explain the example Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply review suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add zarr and torch_dist explanation --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * add subsection Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Update docs/source/checkpoints/intro.rst Co-authored-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix code block Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * formatting Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: Jan Lasek <janek.lasek@gmail.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Malay Nagda <malayn@malayn-mlt.client.nvidia.com>
* Add checkpoints section Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix title Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Add section on ".qnemo" checkpoints (#9503) * Add 'Quantized Checkpoints' section Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Address review comments Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Distributed checkpointing user guide (#9494) * Describe shardings and entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Strategies, optimizers, finalize entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Transformations Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Integration Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add link from intro Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply grammar suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Explain the example Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply review suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add zarr and torch_dist explanation --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * add subsection Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Update docs/source/checkpoints/intro.rst Co-authored-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix code block Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * formatting Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: Jan Lasek <janek.lasek@gmail.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Co-authored-by: Chen Cui <chcui@nvidia.com>
* Add checkpoints section Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Fix title Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * update Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Add section on ".qnemo" checkpoints (NVIDIA#9503) * Add 'Quantized Checkpoints' section Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Address review comments Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Distributed checkpointing user guide (NVIDIA#9494) * Describe shardings and entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Strategies, optimizers, finalize entrypoints Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Transformations Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Integration Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add link from intro Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply grammar suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Explain the example Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Apply review suggestions Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * Add zarr and torch_dist explanation --------- Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> * add subsection Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * Update docs/source/checkpoints/intro.rst Co-authored-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix code block Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * address comments Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * formatting Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * fix Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com> Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: Jan Lasek <janek.lasek@gmail.com> Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>
What does this PR do ?
Add NeMo Dev docs to explain NeMo checkpoints, .nemo .ckpt, regular and distributed.
GitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information