Torchserve 23mt v0.8.0 fix 1 #14
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Update index.md Update to fix a broken link in index.md where the trailing .md is cut off from the management_api.md. Added an anchor link to force the .md to show up. * Update to index.md Update to index.md to fix several links ending in .md that sphinx is breaking. Added anchor links to each link and a corresponding anchor in the affected doc. Tested locally and seems to be working. * Update inference_api.md * Updated typos Fixed typos and updated wordslist.txt * Update wordlist.txt --------- Co-authored-by: sekyonda <7411+sekyonda@users.noreply.ghe.oculus-rep.com> Co-authored-by: lxning <23464292+lxning@users.noreply.github.com>
* Inf2 nightly benchmark * fix linter spellcheck error --------- Co-authored-by: Naman Nandan <namannan@amazon.com>
* fix: kserve fastapi migration issues
* Update index.md Update to fix a broken link in index.md where the trailing .md is cut off from the management_api.md. Added an anchor link to force the .md to show up. * Update to index.md Update to index.md to fix several links ending in .md that sphinx is breaking. Added anchor links to each link and a corresponding anchor in the affected doc. Tested locally and seems to be working. * Update inference_api.md * Updated typos Fixed typos and updated wordslist.txt * Update wordlist.txt * FAQs Updates Updated a couple of broken links on the FAQ site per issue pytorch#2204 * updates to resolve links * Update some links in index Updated some links in index.md to go to the pytorch html page instead of github. This is a nicer fix for the .md sphinx issue --------- Co-authored-by: sekyonda <7411+sekyonda@users.noreply.ghe.oculus-rep.com> Co-authored-by: lxning <23464292+lxning@users.noreply.github.com> Co-authored-by: Ankith Gunapal <agunapal@ischool.Berkeley.edu>
* Add unit test for micro batching * Adds simple implementation for microbatching with coroutines * Added testing to microbatching * Created MBHandler * Added more tests to microbatching handler * Adds configurable parallelism * Enables loading of micro batching parameters through config file * Moved microbatching into example * Moved micro batching test into test/pytest folder * Rewrote micro batching to use threading * Implemented method to update parallelism * Fix and test spin up spin down fo threads * Clean up and comments * More comments * Adds README to micro batching example * Refined readme + added config.yaml * Add config_file * Fix linting error * Fix spell check error * Fix linting error * Move micro_batching.py into ts.utils and use model_yaml_config for configuration * Fix links in README * Moved to handler_utils * remove __all__
Co-authored-by: Ankith Gunapal <agunapal@ischool.Berkeley.edu>
pytorch#2371) Co-authored-by: Naman Nandan <namannan@amazon.com>
* Enable ONNX test in CI
* update cpuinfo * update * lint * lint * lint * install numactl --------- Co-authored-by: Ankith Gunapal <agunapal@ischool.Berkeley.edu> Co-authored-by: Mark Saroufim <marksaroufim@fb.com>
* fix tcp port in model_service_worker.py * update torchrun parameter name * fmt
* update cpuinfo * update * lint * lint * lint * install numactl * update num workers * skip if gpu --------- Co-authored-by: Ankith Gunapal <agunapal@ischool.Berkeley.edu> Co-authored-by: Mark Saroufim <marksaroufim@fb.com>
…h#2374) torch == 2.0.1 torchvision == 0.15.2 torchtext == 0.15.2 torchaudio == 2.0.2 Co-authored-by: Naman Nandan <namannan@amazon.com> Co-authored-by: Ankith Gunapal <agunapal@ischool.Berkeley.edu>
* adding type annotations to functions
* reflect the model config from properties * honor model config in config.properties
Bumps [transformers](https://github.com/huggingface/transformers) from 4.28.1 to 4.30.0. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](huggingface/transformers@v4.28.1...v4.30.0) --- updated-dependencies: - dependency-name: transformers dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ytorch#2350) * add jobQsize * add jobQueueSize in model_config.yaml for mnist_scripted.mar * feature job tickets * rename hasJobTickets to getJobTickets * fmt doc * update doc * fmt doc * fmt doc * update doc
…:pytorch/serve into torchserve-23mt-v0.8.0-fix-1
pypae
approved these changes
Jun 14, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR incorporates the changes from upstream until this commit pytorch@89fbd88
This should fix an issue in v0.8.0 where the batch size of a model could not be set in config.properties anymore. In particular it incorporates this PR here: pytorch#2382
To create this PR I pulled the commit in question with
git pull --no-rebaseinstead of rebasing like last time. I am not sure if this is an issue.I tested and verified that reading properties such as
batchSizenow indeed works, at least with the cpu version. I also verified that with this version, there are indeed multiple batches send to thehandlefunction of the model server handlers.Current docker builds are
textshuttle/pytorch-serve:torchserve-23mt-v0.8.0-fix-1-cpuandtextshuttle/pytorch-serve:torchserve-23mt-v0.8.0-fix-1-gpu.