-
Notifications
You must be signed in to change notification settings - Fork 863
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update large_model_inference.md #2542
Update large_model_inference.md #2542
Conversation
Codecov Report
@@ Coverage Diff @@
## master #2542 +/- ##
=======================================
Coverage 72.64% 72.64%
=======================================
Files 79 79
Lines 3733 3733
Branches 58 58
=======================================
Hits 2712 2712
Misses 1017 1017
Partials 4 4 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
@@ -1,6 +1,13 @@ | |||
# Serving large models with Torchserve | |||
|
|||
This document explain how Torchserve supports large model serving, here large model refers to the models that are not able to fit into one gpu so they need be split in multiple partitions over multiple gpus. | |||
This page is split into the following sections: | |||
- [How it works](#how-it-works) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I remember there was some issue with relative links when we go to pytorch.org so might wanna double check this works, if it does feel free to dismiss this comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pending one query about low_cpu_mem_usage=True
, rest looks fine
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left a suggestion
Description
Adding DeepSpeed MII and Hugging face accelerate information to LMI doc.
Type of change
Please delete options that are not relevant.
Feature/Issue validation/testing
Built pages locally.
Checklist: