Integration test huggingface transfromers torchscript support #433

HamidShojanazeri · 2020-06-05T16:42:03Z

Description

Add integration tests for Huggingface transformers with torchscript/eager mode support and removed previous Huggingface transformers tests.

Fixes #(issue)
Add integration test based on the new handler that supports torchscript.

Feature/Issue validation/testing.

Switch to branch issue_391
cd serve/docker
./build_image.sh
docker run -it --user root pytorch/torchserve:latest /bin/bash
apt-get update
apt-get install -y git wget sudo
git clone https://github.com/pytorch/serve
cd serve
checkout to issue_391
Add tests as shown in the added_test file in the logs
./test/regression_tests.sh staging_0_1_1

Logs
RegressionTest_complete.log
added_test.txt

Checklist:

New and existing unit tests pass locally with these changes?

…hat support torchscript

sagemaker-neo-ci-bot · 2020-06-05T16:52:47Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 33d14d8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-06-05T16:57:28Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 33d14d8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…romers_torchscriptSupport

sagemaker-neo-ci-bot · 2020-06-05T17:17:18Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: cc402ce
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-06-05T17:21:47Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: cc402ce
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…romers_torchscriptSupport

sagemaker-neo-ci-bot · 2020-06-05T18:51:51Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 4f758cb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-06-05T18:56:29Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 4f758cb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…romers_torchscriptSupport

sagemaker-neo-ci-bot · 2020-06-05T20:17:26Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 705062d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-06-05T20:21:24Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 705062d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-06-05T20:50:54Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 705062d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-06-05T20:54:48Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 705062d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

* fixed resnet152 custom handler to work with GPU * issue template * Moving issue templates to root folder per pull request template * adding feature request template * Minor changes related to sections and names. * Update feature_template.md * Update bug_template.md * doc template * minor changes * Minor changes to upload custom model or handler * issue templates (#273) * issue template * Moving issue templates to root folder per pull request template * adding feature request template * Minor changes related to sections and names. * Update feature_template.md * Update bug_template.md * doc template * minor changes * Minor changes to upload custom model or handler Co-authored-by: mycpuorg <mail@mycpu.org> * Fixes #205 - Docker install not finding GPUs (#262) Invokes docker with --runtime=nvidia for GPU Added the option to specify the GPU / specific GPU ids in start.sh Fixed the documentation for start.sh script Fixed the JDK version in Dockerfile.gpu Co-authored-by: Aaqib <maaquib@gmail.com> * Fix pylint issue (#163) * pylint fixes and code cleanup * UT fixes, pylint updates per latest version and updated sanity script to run pylint * fixed pylint and wrapt installation issue * freeze pylint version to 2.4.4 due to bug in version 2.5 * removed duplicate statement to install pylint Co-authored-by: Aaqib <maaquib@gmail.com> * Updated exception handling for worker threads shutdown. (#172) * updated code to now log errors only in case when exception is not genreated by unregister api call * java formatting * fixed illigalstateexception * removed the unrequired handling for illegal monitor state exception Co-authored-by: dhaniram kshirsagar <dhaniram_kshirsagar@persistent.com> Co-authored-by: Aaqib <maaquib@gmail.com> * changes for Homebrew issue 98 (#126) * changes for Homebrew issue 98 * adding changes for path in benchmark.py * CI Build corrected * benchmark changes corrected path * changed PATH for homebrew * Update Readme Made changes for homebrew installation and instruction for GPU based machine instruction for install_dependencies.sh * Updated Readme.md removed unnecessory lines * corrected indentation corrected indentation for new lines added * Update Readme.md fil added separate commands for CPU and GPU based Co-authored-by: Ubuntu <ubuntu@ip-172-31-35-42.ec2.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-42-124.ec2.internal> Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-67.ec2.internal> Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * added changes to clean model loaded through URL from model store on unregisted * added UTs * Moving the plugins SDK to org.pytorch. Added plugins example. (#261) * Moved plugins SDK to org.pytorch * Updating the repo for publishing plugins SDK * Enabled bintray maven repo temporarily * Reset tests * Added GPG plugin to pom file to sign the plugins SDK * Update the SDK version * Removed bintray repo * Modify the buildspec to not deploy the SDK Co-authored-by: Vamshidhar Dantu <dantu@amazon.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: mycpuorg <mail@mycpu.org> * Added model zoo (#299) Co-authored-by: mycpuorg <mail@mycpu.org> Co-authored-by: Brad Heintz <bradheintz@fb.com> * Removed snapshot dependency on model (#152) * Snapshot is now independent of model * fixed checkstyle issue * fixed formatting * doc review: configuration and custom_service * addressed review comments * fixed incorrect merge * fixed review comments and corrected the error messages * adding remaining fixes for string constants Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaron Markham <markhama@amazon.com> Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> * Regression Tests Suite - AWS Code Build Infra - Initial Commit for #57 (#285) * Regression Tests Suite - AWS Code Build Infra - Initial Commit for #57 * Script to clone Pytorch Repo & Run Post Man Scripts * Basic Tests for Inference & Managment APIs * AWS Code Build buildspec.yml * Refactor postman tests, add empty postman collection for todos * Add a readme for adding tests * Add more management tests * Fix typo * Snapshot Regression tests - #57 * Fix typo * Comment out snapshot test for now * Weird typos are showing up, try saving everything * Need to manually save every request in postman... * When reregister don't use s3 * More workarounds for rereg * Fix test messages * Remove test for empty reqs * Change get to post * Fix pytest for subprocess - #57 * Enable pytest - #57 * Inference API Testcases / Rename Management Testcases #57 * Wipe Model MARs in Modelstore between tests #54 * Fix Inference Endpoint Execution #54 * Fix file paths - #57 * Fix Initial Workers #57 * Fix Inference Assertions #57 * Fix DenseNet Inference #57 * Fix Densenet Register #54 * Redirect Logs to File #54 * Fix Test Execution Log #54 * README, Build Badge #57 * Format ReadMe #57 * Update README - Add Instruction to add new tests / Run tests locally * Update README * Update branch for build Codebuild Badge Co-authored-by: alexwong <11878166+alexwong@users.noreply.github.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: mycpuorg <mail@mycpu.org> Co-authored-by: Ubuntu <ubuntu@ip-172-31-12-178.us-west-2.compute.internal> * Issue 312 install from src (#322) * documentation: fixed issues in installation and quick start in README.md * documentation: Reverting conda related doc changes from PR#286 Fixes #297 * remove minus y (#311) update pip install instructions * Update instructions for installation from source This commit aims to address Issue #312 Tested on a fresh EC2 P3.2 instance: * Followed instructions under https://github.com/pytorch/serve#install-torchserve-for-development * Started the server manually from command line * Ran a ping query Received a response from server with "Healthy" status Signed-off-by: Manoj Rao <mail@mycpu.org> * add separate files for installation one each for linux and macos * update README and remove relative path from script * Install script - Dont uninstall if not previously installed - #132 * explicitly build the source code with gradle prior to install * remove upgrade instructions in favor of clean reinstall from source * update instructions to contain the correct script name Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaqib Ansari <maaquib@gmail.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaron Markham <markhama@amazon.com> Co-authored-by: Manoj Rao <mail@mycpu.org> Co-authored-by: Ubuntu <ubuntu@ip-172-31-0-61.us-west-2.compute.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-7-197.us-west-2.compute.internal> * fix for macos script (#338) * Move Issue Template to .github dir * Updated sanity script to use install_from_src_ubuntu script (#350) * updated sanity script to use install_from_src_ubuntu script * added java version check * fixed issue with if statement * fixed variable name * fixed repeated cd command * updated artifacts * removed artifacts from buildspec * Corrected HTTP response code for Model version does not exists exception (#282) * issue 196 patch * Added openapi missing responses. - incorporated review comments * updated method signature after resolving merge conflicts Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Shivam Shriwas <shivamshriwas21@gmail.com> Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Corrected paths in command and added notes (#354) Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Refactored UT for better reporting (#159) * refactored UTs * upgraded testng version and updated testcase names * added testng.xml to handle sequential execution of test classes * added waveglow example * Updated readme for waveglow example * fixed failing UTs after merging master * removed unintentionally pushed waveglow example * fixed failing test case * moved set-default to utils and added couple of more add cases for default version * java formatting * corrected set default negative test case * fixed java formatting Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Huggingface Transformers generalization (#302) * Scripts to seriliaze a Bert model using jit.trace and a handler for Bert model * modified to include mapping of predicitons and decoding received text * move back the original text * moved the index_to_name to bert folder * Transformers/ Bert serving with Torchserve readme * deleting an unused dictionary * just cleaning some of the commented out parts * cleaning up * Cleaning up * uncommenting the logging and cleaning the comments * Download and save the checkpoints, config file and vocab of the BERT from Transformers for demonstration purposes * mapping of predicitons to labels * a custom handler for transformers. * README. * sample of input text. * modifying the descriptions * modifying the descriptions * adding comments * adding commnets * Fromatting modifications * modified comments * added the citations * adding hugginface as the type of transformer model we are referring to * renaming the folder to huggingface_transformers * Removing the bert folder * simple modification * modifed the script to use a setup_config.json for loading and saving models * added the section for transfomer_handler_generalized * the generalized handler * set_up config intialization * edited for transfomer generalization * modified to use just input_ids as inputs for models * modified for question answering * modified to include the number of labels * corrected typos * modifed variable names * edited for clarity * modifed variable names, simplified the initlialize in the handler class, removed the warnning from QA case * modifed values for QA * modifed the variable names * added comments * modified to add clarity * add comment in preprocessing of QA * added warning for the case mode is not selected properly and combined constructor and initlaize fucntion * clean up * clean up * combined constructor and initliaze func * combined constructor and initliaze func * Separated the constructor and initliaze func * Modified to remove the classification part and update with generalized version * resolved the coflict and separated the costructor and initliaze func * removed the handler for classification case as the generalized version handles the case Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: mycpuorg <mail@mycpu.org> * added waveglow example (#258) * added waveglow example * Updated readme for waveglow example * enhanced example to remove runtime checkpoint download * updated handler to return the audio file content instead of dumping it to tmp dir * minor change to handler * added pipefail * added python snippet to run inference * updated readme with know issue with the example Co-authored-by: Aaqib <maaquib@gmail.com> * made build suite compatible with latest gradle (#348) Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaqib <maaquib@gmail.com> * moved env section after context * Deleting redundant ISSUE_TEMPLATE at root dir * Fix typo in scripts/install_from_src_macos * Corrected OpenApi description (#328) * Correct query parameter name (Registration Api)in OpenApi description * removed non existing path parameter Co-authored-by: Aaqib <maaquib@gmail.com> * Regression suite fixes and added test cases (#344) * added test model zoo test cases updated regression script * Updated Readme * update -y flag for install * Updated docker command as per new imagename and tag generated in stagging branch * Fix sudo usage in regression test script * - addressed review comments - reverted changes related to sudo in regression script. * added https test collection * - updated failed test cases on staging branch - added environment and updated newman cmd with --insecure for https * updated script * updated pytest * update curl check for serve up * Revert "update curl check for serve up" This reverts commit 0047eae. * Revert "update curl check for serve up" This reverts commit 0047eae. - update model zoo - added resnet test case * added workers to resnet model * Update readme to run test suite on desired branch * enhanced inference test cases to driven by datafile * Updated readme Co-authored-by: Dhanasekar Karuppasamy <dhanak@amazon.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Fix typo in mnist readme (#330) Co-authored-by: Aaqib <maaquib@gmail.com> * Fixed typo in ab installation. (#327) * documentation: fixed issues in installation and quick start in README.md * documentation: Reverting conda related doc changes from PR#286 Fixes #297 * remove minus y (#311) update pip install instructions * Fixed typo in ab installation. Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaqib Ansari <maaquib@gmail.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaron Markham <markhama@amazon.com> * Setup Nightly Build for TorchServe (#371) * Add Branch Flag for Nightly CodeBuild #57 * Update ENV Variable for Nightly Build Name * Nigtly Build - Fix FileName for Build Artifacts * Update README for accessing test logs * Whitspace changes * Build new images for build - README Instruction * Fix for intermittent gradle build failure (#352) * Fix for intermittent null pointer exception * Fixed comments Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> Co-authored-by: Aaqib <maaquib@gmail.com> * documentation: Remove redundant curl options in example (#336) * documentation: fixed issues in installation and quick start in README.md * documentation: Reverting conda related doc changes from PR#286 Fixes #297 * remove minus y (#311) update pip install instructions * fix-doc * fix remaining "-X POST"s Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaqib Ansari <maaquib@gmail.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaron Markham <markhama@amazon.com> * Corrected port number validation range (#304) * Corrected port number validation range * added UT for issue 291 * added ut dependency file formatting * correct param in config_test_env.properties * removed incorrect flags from test case config files * added negative test cases * removed properties file and changes in test case Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> * Fixes version number reporting for TorchServe (#360) * Correct version and added vesion.txt for getting version info * fixed description test cases and build issue * Added snapshot test fixes and resolved relative path issue for version.txt * python formatting * added version.txt in MANIFEST.in * addressed review comments * added version.txt for model archiver as well - addressed review comments * -fixed python UT -added ts python UT run in sanity script * removed blank lines from snapshot files Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> * fixes installation script exits without error if no java present on machine (#376) * fixes installation script exits without error if no java present on machine * added changes for macos script Co-authored-by: Aaqib <maaquib@gmail.com> * Dockerfile enhancements (#308) * dockerfile now points to latest torchserve on pypi and added separate files for dev * Minor changes * Made build and run commands consistent * Fixing formatting * Consistency issue * Enhanced Dockerfile for binary per #178 and other misc. * Fixed gpu related issues in start script * Corrected the heading font * Added mar creation steps from container * added buildkit support for dev docker files * Fix for buildkit * Restoring psutil and future as absence leads intermittent hangged build * Fixed review comments for doc * Fixed typo Co-authored-by: Ubuntu <ubuntu@ip-172-31-64-54.ec2.internal> Co-authored-by: Aaqib <maaquib@gmail.com> * Updated configuration.md for missing config properties (#337) * Updated configuration.md for missing parameters * changes TS_MODEL_SNAPSHOT to MODEL_SNAPSHOT - removes ambiguity - follows naming convention * addressed review comments. * removed debug property * updated configuration.md Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * Adding missing document links to README (#379) * Maven tests should output only errors (#326) Co-authored-by: Vamshidhar Dantu <dantu@amazon.com> Co-authored-by: Aaqib <maaquib@gmail.com> * fixed code for build failure in python virtual environment (#341) * fixed code for build failure in python virtual environment * added exception missed on resolving conflicts Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Vamshidhar Dantu <36211508+vdantu@users.noreply.github.com> * updated object detector handler to use only default cuda device for torchvision version less then 0.6 (#265) * updated object detector handler to use default cuda device for torchvision version 0.5.0 * updated version check logic * Updated documentation for torchvision version dependency * enhanced sanity script to validate all default handlers * fixed sanity script * added torchtext install step in sanity script * fixed issue related to inference after restart with snapshot. * moved torchtext to codebuild docker files * added more log statements * enhanced run inference function * fixed register model function * added unregister model function * incorporated code review comments * added back inference call for snapshot test * fixed if block Co-authored-by: Aaqib <maaquib@gmail.com> * Adding a barebones ec2 template to run torchserve on ubuntu (#301) * Adding a barebones ec2 template to run torchserve on ubuntu * Adding VPC with Private subnet * Address review comments #1 * Address review comments #2 * Rebase for #365 (#389) * Updated README.md for #364 See #364 for a deep description of the problem * updated README.md * updated README.md Co-authored-by: Francesco Saverio Zuppichini <zuppif@usi.ch> * Document the docker settings for running in production mode (#388) * Docker Production Options #234 * Docker Production Options - Typo Fixes / Fit & Finish #234 * Typos - Docker Docs #234 * grammar check - docker/README.md Co-authored-by: Aaqib <maaquib@gmail.com> * grammar check - docker/README.md Co-authored-by: Aaqib <maaquib@gmail.com> * grammar check - docker/README.md Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * Huggingface transformers integration test (#382) * Add Huggingface transformers example integration tests * Add transformers intallation * Fix a missing bracket * move transformers installation to chain installations Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> * Updated docker README.md (#401) * Update README.md * Fixed cd command * Made source cloning steps explicit!! * Fixed online editing mistake Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> * Fixed response status message for inferencing on model with zero workers (#305) * Changes for issue_29 * Changed error Messages * changed message for scalling request * changed defaultVersion string * Corrected error message * changed the code for best practice * resolved review comment for version * updated management api doc as per new responses in worker scale api Co-authored-by: Ubuntu <ubuntu@ip-172-31-80-124.ec2.internal> Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> Co-authored-by: Shivam Shriwas <shivamshriwas21@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * made model server port tests independent (#396) * Added backend profiling and fixed benchmark's install dependency scripts (#402) * added backend profiling * updated install dependency scripts to make them jmeter version independent * updated readme * Fix benchmarks/README typo and JMeter versioning issue * updated documentation Co-authored-by: Aaqib Ansari <maaquib@gmail.com> * Add sphinx directory and conf.py (#274) * Add sphinx folder that contains files for docs build * Delete .DS_Store * Delete .DS_Store * Add conf.py in docs/ * documentation: fixed issues in installation and quick start in README.md * documentation: Reverting conda related doc changes from PR#286 Fixes #297 * remove minus y (#311) update pip install instructions * Update to the latest version of sphinx * Add recommonmark to extensions Add recommonmark to extensions so sphinx project has markdown support. Configuration details here: https://www.sphinx-doc.org/en/master/usage/markdown.html * Update source suffix to use latest recommonmark syntax Updating to the syntax found here: https://www.sphinx-doc.org/en/master/usage/markdown.html Co-authored-by: Aaron Markham <markhama@amazon.com> * Remove old recommonmark parser syntax * Pin sphinx version to work with latest version of recommonmark Co-authored-by: Aaron Markham <markhama@amazon.com> * Pin version latest version recommonmark Co-authored-by: Aaron Markham <markhama@amazon.com> * Delete theme file generated during build * de-duplicate install package of psutil (#329) The Dockerfile of GPU install package psutil duplicated. * Remove pytorch-sphinx-theme files * Update to match latest staging_0_1_1 * Update docker directory to match staging_0_1_1 branch * Delete .DS_Store * Add sphinxcontrib.katex and torch Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaqib Ansari <maaquib@gmail.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaron Markham <markhama@amazon.com> Co-authored-by: Mario Cho <hephaex@gmail.com> Co-authored-by: mycpuorg <mail@mycpu.org> * Image classifier example readme fixes (#405) * Changed the model name in torch.jit.save to be consistent with docs * Remove extra handler flag and fix the torchservr start * fix the doc * fix the doc Co-authored-by: mycpuorg <mail@mycpu.org> * Porting memory fix changes to TorchServe (#422) Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Benchmark documentation fix (#425) * Fix Benchmark Documentation * Documentaion fixes * .circleci: Add initial circleci configuration (#431) Signed-off-by: Eli Uriegas <eliuriegas@fb.com> * Torchscript support huggingface transfrmers (#421) * Add support for torchscript * Add support for torchscript * Add key values for torchscript support * Add Torchscript suport * Add Torchscript support * Add Torchscript support in loading the models * keep the example to QA * modify to reflect Torchscript example? * Add torchscript support guides * Add fintued BERT for QA * Add QA artifacts * Add seqenece classification artifacts * Token classification artifacts * Correct typos * Add max length to tokenizer * clean up * enhance the doc * change setup to seq classification * Add detail to torchscript support * set the model for QA * Add torchscript bert seq classification example * Add bert seq classification example * Modify the packaging model command * Modify torchscript packaging command * clean up * clean up * correct spacing between list items Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Integration test huggingface transfromers torchscript support (#433) * made model server port tests independent * Add integration tests for Huggingface transfromers with new handler that support torchscript Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: mycpuorg <mail@mycpu.org> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Apachebench benchmark - cleanup script / documentation * Revert "Apachebench benchmark - cleanup script / documentation" This reverts commit c8f8682. * Documentation updates for clarity on Eager mode / Torch scripted models (#439) * Model Zoo updates for Eager Mode / Torchscripted models * Update examples/README.md Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * Apachebench benchmarking - Script / Documentation updates (#438) * Apachebench benchmarking script - Script / Documentation updates * Stop TS after benchmark completion. Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: dhaniram-kshirsagar <26479924+dhaniram-kshirsagar@users.noreply.github.com> Co-authored-by: dhaniram kshirsagar <dhaniram_kshirsagar@persistent.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: mycpuorg <mail@mycpu.org> Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> Co-authored-by: mjpsl <57522192+mjpsl@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-35-42.ec2.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-42-124.ec2.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-67.ec2.internal> Co-authored-by: Vamshidhar Dantu <36211508+vdantu@users.noreply.github.com> Co-authored-by: Vamshidhar Dantu <dantu@amazon.com> Co-authored-by: Brad Heintz <bradheintz@fb.com> Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaron Markham <markhama@amazon.com> Co-authored-by: alexwong <11878166+alexwong@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-12-178.us-west-2.compute.internal> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-0-61.us-west-2.compute.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-7-197.us-west-2.compute.internal> Co-authored-by: dkasar-aws <57526278+dkasar-aws@users.noreply.github.com> Co-authored-by: Shivam Shriwas <shivamshriwas21@gmail.com> Co-authored-by: Hamid Shojanazeri <hamid.nazeri2010@gmail.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-14-91.us-west-2.compute.internal> Co-authored-by: shivamshriwas <44968110+shivamshriwas@users.noreply.github.com> Co-authored-by: Takayoshi Nishida <takayoshi.nishida@gmail.com> Co-authored-by: alvations <alvations@gmail.com> Co-authored-by: Tzu-Wei Huang <huang.dexter@gmail.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-64-54.ec2.internal> Co-authored-by: Francesco Saverio Zuppichini <zuppif@usi.ch> Co-authored-by: Ubuntu <ubuntu@ip-172-31-80-124.ec2.internal> Co-authored-by: Jessica Lin <jplin@fb.com> Co-authored-by: Mario Cho <hephaex@gmail.com> Co-authored-by: Eli Uriegas <1700823+seemethere@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-78-248.ec2.internal>

* fixed resnet152 custom handler to work with GPU * issue template * Moving issue templates to root folder per pull request template * adding feature request template * Minor changes related to sections and names. * Update feature_template.md * Update bug_template.md * doc template * minor changes * Minor changes to upload custom model or handler * issue templates (#273) * issue template * Moving issue templates to root folder per pull request template * adding feature request template * Minor changes related to sections and names. * Update feature_template.md * Update bug_template.md * doc template * minor changes * Minor changes to upload custom model or handler Co-authored-by: mycpuorg <mail@mycpu.org> * Fixes #205 - Docker install not finding GPUs (#262) Invokes docker with --runtime=nvidia for GPU Added the option to specify the GPU / specific GPU ids in start.sh Fixed the documentation for start.sh script Fixed the JDK version in Dockerfile.gpu Co-authored-by: Aaqib <maaquib@gmail.com> * Fix pylint issue (#163) * pylint fixes and code cleanup * UT fixes, pylint updates per latest version and updated sanity script to run pylint * fixed pylint and wrapt installation issue * freeze pylint version to 2.4.4 due to bug in version 2.5 * removed duplicate statement to install pylint Co-authored-by: Aaqib <maaquib@gmail.com> * Updated exception handling for worker threads shutdown. (#172) * updated code to now log errors only in case when exception is not genreated by unregister api call * java formatting * fixed illigalstateexception * removed the unrequired handling for illegal monitor state exception Co-authored-by: dhaniram kshirsagar <dhaniram_kshirsagar@persistent.com> Co-authored-by: Aaqib <maaquib@gmail.com> * changes for Homebrew issue 98 (#126) * changes for Homebrew issue 98 * adding changes for path in benchmark.py * CI Build corrected * benchmark changes corrected path * changed PATH for homebrew * Update Readme Made changes for homebrew installation and instruction for GPU based machine instruction for install_dependencies.sh * Updated Readme.md removed unnecessory lines * corrected indentation corrected indentation for new lines added * Update Readme.md fil added separate commands for CPU and GPU based Co-authored-by: Ubuntu <ubuntu@ip-172-31-35-42.ec2.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-42-124.ec2.internal> Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-67.ec2.internal> Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * added changes to clean model loaded through URL from model store on unregisted * added UTs * Moving the plugins SDK to org.pytorch. Added plugins example. (#261) * Moved plugins SDK to org.pytorch * Updating the repo for publishing plugins SDK * Enabled bintray maven repo temporarily * Reset tests * Added GPG plugin to pom file to sign the plugins SDK * Update the SDK version * Removed bintray repo * Modify the buildspec to not deploy the SDK Co-authored-by: Vamshidhar Dantu <dantu@amazon.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: mycpuorg <mail@mycpu.org> * Added model zoo (#299) Co-authored-by: mycpuorg <mail@mycpu.org> Co-authored-by: Brad Heintz <bradheintz@fb.com> * Removed snapshot dependency on model (#152) * Snapshot is now independent of model * fixed checkstyle issue * fixed formatting * doc review: configuration and custom_service * addressed review comments * fixed incorrect merge * fixed review comments and corrected the error messages * adding remaining fixes for string constants Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaron Markham <markhama@amazon.com> Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> * Regression Tests Suite - AWS Code Build Infra - Initial Commit for #57 (#285) * Regression Tests Suite - AWS Code Build Infra - Initial Commit for #57 * Script to clone Pytorch Repo & Run Post Man Scripts * Basic Tests for Inference & Managment APIs * AWS Code Build buildspec.yml * Refactor postman tests, add empty postman collection for todos * Add a readme for adding tests * Add more management tests * Fix typo * Snapshot Regression tests - #57 * Fix typo * Comment out snapshot test for now * Weird typos are showing up, try saving everything * Need to manually save every request in postman... * When reregister don't use s3 * More workarounds for rereg * Fix test messages * Remove test for empty reqs * Change get to post * Fix pytest for subprocess - #57 * Enable pytest - #57 * Inference API Testcases / Rename Management Testcases #57 * Wipe Model MARs in Modelstore between tests #54 * Fix Inference Endpoint Execution #54 * Fix file paths - #57 * Fix Initial Workers #57 * Fix Inference Assertions #57 * Fix DenseNet Inference #57 * Fix Densenet Register #54 * Redirect Logs to File #54 * Fix Test Execution Log #54 * README, Build Badge #57 * Format ReadMe #57 * Update README - Add Instruction to add new tests / Run tests locally * Update README * Update branch for build Codebuild Badge Co-authored-by: alexwong <11878166+alexwong@users.noreply.github.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: mycpuorg <mail@mycpu.org> Co-authored-by: Ubuntu <ubuntu@ip-172-31-12-178.us-west-2.compute.internal> * Issue 312 install from src (#322) * documentation: fixed issues in installation and quick start in README.md * documentation: Reverting conda related doc changes from PR#286 Fixes #297 * remove minus y (#311) update pip install instructions * Update instructions for installation from source This commit aims to address Issue #312 Tested on a fresh EC2 P3.2 instance: * Followed instructions under https://github.com/pytorch/serve#install-torchserve-for-development * Started the server manually from command line * Ran a ping query Received a response from server with "Healthy" status Signed-off-by: Manoj Rao <mail@mycpu.org> * add separate files for installation one each for linux and macos * update README and remove relative path from script * Install script - Dont uninstall if not previously installed - #132 * explicitly build the source code with gradle prior to install * remove upgrade instructions in favor of clean reinstall from source * update instructions to contain the correct script name Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaqib Ansari <maaquib@gmail.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaron Markham <markhama@amazon.com> Co-authored-by: Manoj Rao <mail@mycpu.org> Co-authored-by: Ubuntu <ubuntu@ip-172-31-0-61.us-west-2.compute.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-7-197.us-west-2.compute.internal> * fix for macos script (#338) * Move Issue Template to .github dir * Updated sanity script to use install_from_src_ubuntu script (#350) * updated sanity script to use install_from_src_ubuntu script * added java version check * fixed issue with if statement * fixed variable name * fixed repeated cd command * updated artifacts * removed artifacts from buildspec * Corrected HTTP response code for Model version does not exists exception (#282) * issue 196 patch * Added openapi missing responses. - incorporated review comments * updated method signature after resolving merge conflicts Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Shivam Shriwas <shivamshriwas21@gmail.com> Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Corrected paths in command and added notes (#354) Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Refactored UT for better reporting (#159) * refactored UTs * upgraded testng version and updated testcase names * added testng.xml to handle sequential execution of test classes * added waveglow example * Updated readme for waveglow example * fixed failing UTs after merging master * removed unintentionally pushed waveglow example * fixed failing test case * moved set-default to utils and added couple of more add cases for default version * java formatting * corrected set default negative test case * fixed java formatting Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Huggingface Transformers generalization (#302) * Scripts to seriliaze a Bert model using jit.trace and a handler for Bert model * modified to include mapping of predicitons and decoding received text * move back the original text * moved the index_to_name to bert folder * Transformers/ Bert serving with Torchserve readme * deleting an unused dictionary * just cleaning some of the commented out parts * cleaning up * Cleaning up * uncommenting the logging and cleaning the comments * Download and save the checkpoints, config file and vocab of the BERT from Transformers for demonstration purposes * mapping of predicitons to labels * a custom handler for transformers. * README. * sample of input text. * modifying the descriptions * modifying the descriptions * adding comments * adding commnets * Fromatting modifications * modified comments * added the citations * adding hugginface as the type of transformer model we are referring to * renaming the folder to huggingface_transformers * Removing the bert folder * simple modification * modifed the script to use a setup_config.json for loading and saving models * added the section for transfomer_handler_generalized * the generalized handler * set_up config intialization * edited for transfomer generalization * modified to use just input_ids as inputs for models * modified for question answering * modified to include the number of labels * corrected typos * modifed variable names * edited for clarity * modifed variable names, simplified the initlialize in the handler class, removed the warnning from QA case * modifed values for QA * modifed the variable names * added comments * modified to add clarity * add comment in preprocessing of QA * added warning for the case mode is not selected properly and combined constructor and initlaize fucntion * clean up * clean up * combined constructor and initliaze func * combined constructor and initliaze func * Separated the constructor and initliaze func * Modified to remove the classification part and update with generalized version * resolved the coflict and separated the costructor and initliaze func * removed the handler for classification case as the generalized version handles the case Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: mycpuorg <mail@mycpu.org> * added waveglow example (#258) * added waveglow example * Updated readme for waveglow example * enhanced example to remove runtime checkpoint download * updated handler to return the audio file content instead of dumping it to tmp dir * minor change to handler * added pipefail * added python snippet to run inference * updated readme with know issue with the example Co-authored-by: Aaqib <maaquib@gmail.com> * made build suite compatible with latest gradle (#348) Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaqib <maaquib@gmail.com> * moved env section after context * Deleting redundant ISSUE_TEMPLATE at root dir * Fix typo in scripts/install_from_src_macos * Corrected OpenApi description (#328) * Correct query parameter name (Registration Api)in OpenApi description * removed non existing path parameter Co-authored-by: Aaqib <maaquib@gmail.com> * Regression suite fixes and added test cases (#344) * added test model zoo test cases updated regression script * Updated Readme * update -y flag for install * Updated docker command as per new imagename and tag generated in stagging branch * Fix sudo usage in regression test script * - addressed review comments - reverted changes related to sudo in regression script. * added https test collection * - updated failed test cases on staging branch - added environment and updated newman cmd with --insecure for https * updated script * updated pytest * update curl check for serve up * Revert "update curl check for serve up" This reverts commit 0047eae. * Revert "update curl check for serve up" This reverts commit 0047eae. - update model zoo - added resnet test case * added workers to resnet model * Update readme to run test suite on desired branch * enhanced inference test cases to driven by datafile * Updated readme Co-authored-by: Dhanasekar Karuppasamy <dhanak@amazon.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Fix typo in mnist readme (#330) Co-authored-by: Aaqib <maaquib@gmail.com> * Fixed typo in ab installation. (#327) * documentation: fixed issues in installation and quick start in README.md * documentation: Reverting conda related doc changes from PR#286 Fixes #297 * remove minus y (#311) update pip install instructions * Fixed typo in ab installation. Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaqib Ansari <maaquib@gmail.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaron Markham <markhama@amazon.com> * Setup Nightly Build for TorchServe (#371) * Add Branch Flag for Nightly CodeBuild #57 * Update ENV Variable for Nightly Build Name * Nigtly Build - Fix FileName for Build Artifacts * Update README for accessing test logs * Whitspace changes * Build new images for build - README Instruction * Fix for intermittent gradle build failure (#352) * Fix for intermittent null pointer exception * Fixed comments Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> Co-authored-by: Aaqib <maaquib@gmail.com> * documentation: Remove redundant curl options in example (#336) * documentation: fixed issues in installation and quick start in README.md * documentation: Reverting conda related doc changes from PR#286 Fixes #297 * remove minus y (#311) update pip install instructions * fix-doc * fix remaining "-X POST"s Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaqib Ansari <maaquib@gmail.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaron Markham <markhama@amazon.com> * Corrected port number validation range (#304) * Corrected port number validation range * added UT for issue 291 * added ut dependency file formatting * correct param in config_test_env.properties * removed incorrect flags from test case config files * added negative test cases * removed properties file and changes in test case Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> * Fixes version number reporting for TorchServe (#360) * Correct version and added vesion.txt for getting version info * fixed description test cases and build issue * Added snapshot test fixes and resolved relative path issue for version.txt * python formatting * added version.txt in MANIFEST.in * addressed review comments * added version.txt for model archiver as well - addressed review comments * -fixed python UT -added ts python UT run in sanity script * removed blank lines from snapshot files Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> * fixes installation script exits without error if no java present on machine (#376) * fixes installation script exits without error if no java present on machine * added changes for macos script Co-authored-by: Aaqib <maaquib@gmail.com> * Dockerfile enhancements (#308) * dockerfile now points to latest torchserve on pypi and added separate files for dev * Minor changes * Made build and run commands consistent * Fixing formatting * Consistency issue * Enhanced Dockerfile for binary per #178 and other misc. * Fixed gpu related issues in start script * Corrected the heading font * Added mar creation steps from container * added buildkit support for dev docker files * Fix for buildkit * Restoring psutil and future as absence leads intermittent hangged build * Fixed review comments for doc * Fixed typo Co-authored-by: Ubuntu <ubuntu@ip-172-31-64-54.ec2.internal> Co-authored-by: Aaqib <maaquib@gmail.com> * Updated configuration.md for missing config properties (#337) * Updated configuration.md for missing parameters * changes TS_MODEL_SNAPSHOT to MODEL_SNAPSHOT - removes ambiguity - follows naming convention * addressed review comments. * removed debug property * updated configuration.md Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * Adding missing document links to README (#379) * Maven tests should output only errors (#326) Co-authored-by: Vamshidhar Dantu <dantu@amazon.com> Co-authored-by: Aaqib <maaquib@gmail.com> * fixed code for build failure in python virtual environment (#341) * fixed code for build failure in python virtual environment * added exception missed on resolving conflicts Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Vamshidhar Dantu <36211508+vdantu@users.noreply.github.com> * updated object detector handler to use only default cuda device for torchvision version less then 0.6 (#265) * updated object detector handler to use default cuda device for torchvision version 0.5.0 * updated version check logic * Updated documentation for torchvision version dependency * enhanced sanity script to validate all default handlers * fixed sanity script * added torchtext install step in sanity script * fixed issue related to inference after restart with snapshot. * moved torchtext to codebuild docker files * added more log statements * enhanced run inference function * fixed register model function * added unregister model function * incorporated code review comments * added back inference call for snapshot test * fixed if block Co-authored-by: Aaqib <maaquib@gmail.com> * Adding a barebones ec2 template to run torchserve on ubuntu (#301) * Adding a barebones ec2 template to run torchserve on ubuntu * Adding VPC with Private subnet * Address review comments #1 * Address review comments #2 * Rebase for #365 (#389) * Updated README.md for #364 See #364 for a deep description of the problem * updated README.md * updated README.md Co-authored-by: Francesco Saverio Zuppichini <zuppif@usi.ch> * Document the docker settings for running in production mode (#388) * Docker Production Options #234 * Docker Production Options - Typo Fixes / Fit & Finish #234 * Typos - Docker Docs #234 * grammar check - docker/README.md Co-authored-by: Aaqib <maaquib@gmail.com> * grammar check - docker/README.md Co-authored-by: Aaqib <maaquib@gmail.com> * grammar check - docker/README.md Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * Huggingface transformers integration test (#382) * Add Huggingface transformers example integration tests * Add transformers intallation * Fix a missing bracket * move transformers installation to chain installations Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> * Updated docker README.md (#401) * Update README.md * Fixed cd command * Made source cloning steps explicit!! * Fixed online editing mistake Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> * Fixed response status message for inferencing on model with zero workers (#305) * Changes for issue_29 * Changed error Messages * changed message for scalling request * changed defaultVersion string * Corrected error message * changed the code for best practice * resolved review comment for version * updated management api doc as per new responses in worker scale api Co-authored-by: Ubuntu <ubuntu@ip-172-31-80-124.ec2.internal> Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> Co-authored-by: Shivam Shriwas <shivamshriwas21@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * made model server port tests independent (#396) * Added backend profiling and fixed benchmark's install dependency scripts (#402) * added backend profiling * updated install dependency scripts to make them jmeter version independent * updated readme * Fix benchmarks/README typo and JMeter versioning issue * updated documentation Co-authored-by: Aaqib Ansari <maaquib@gmail.com> * Add sphinx directory and conf.py (#274) * Add sphinx folder that contains files for docs build * Delete .DS_Store * Delete .DS_Store * Add conf.py in docs/ * documentation: fixed issues in installation and quick start in README.md * documentation: Reverting conda related doc changes from PR#286 Fixes #297 * remove minus y (#311) update pip install instructions * Update to the latest version of sphinx * Add recommonmark to extensions Add recommonmark to extensions so sphinx project has markdown support. Configuration details here: https://www.sphinx-doc.org/en/master/usage/markdown.html * Update source suffix to use latest recommonmark syntax Updating to the syntax found here: https://www.sphinx-doc.org/en/master/usage/markdown.html Co-authored-by: Aaron Markham <markhama@amazon.com> * Remove old recommonmark parser syntax * Pin sphinx version to work with latest version of recommonmark Co-authored-by: Aaron Markham <markhama@amazon.com> * Pin version latest version recommonmark Co-authored-by: Aaron Markham <markhama@amazon.com> * Delete theme file generated during build * de-duplicate install package of psutil (#329) The Dockerfile of GPU install package psutil duplicated. * Remove pytorch-sphinx-theme files * Update to match latest staging_0_1_1 * Update docker directory to match staging_0_1_1 branch * Delete .DS_Store * Add sphinxcontrib.katex and torch Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaqib Ansari <maaquib@gmail.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaron Markham <markhama@amazon.com> Co-authored-by: Mario Cho <hephaex@gmail.com> Co-authored-by: mycpuorg <mail@mycpu.org> * fix for response time out issue during build * Image classifier example readme fixes (#405) * Changed the model name in torch.jit.save to be consistent with docs * Remove extra handler flag and fix the torchservr start * fix the doc * fix the doc Co-authored-by: mycpuorg <mail@mycpu.org> * Porting memory fix changes to TorchServe (#422) Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Benchmark documentation fix (#425) * Fix Benchmark Documentation * Documentaion fixes * .circleci: Add initial circleci configuration (#431) Signed-off-by: Eli Uriegas <eliuriegas@fb.com> * Torchscript support huggingface transfrmers (#421) * Add support for torchscript * Add support for torchscript * Add key values for torchscript support * Add Torchscript suport * Add Torchscript support * Add Torchscript support in loading the models * keep the example to QA * modify to reflect Torchscript example? * Add torchscript support guides * Add fintued BERT for QA * Add QA artifacts * Add seqenece classification artifacts * Token classification artifacts * Correct typos * Add max length to tokenizer * clean up * enhance the doc * change setup to seq classification * Add detail to torchscript support * set the model for QA * Add torchscript bert seq classification example * Add bert seq classification example * Modify the packaging model command * Modify torchscript packaging command * clean up * clean up * correct spacing between list items Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Integration test huggingface transfromers torchscript support (#433) * made model server port tests independent * Add integration tests for Huggingface transfromers with new handler that support torchscript Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: mycpuorg <mail@mycpu.org> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Apachebench benchmark - cleanup script / documentation * Revert "Apachebench benchmark - cleanup script / documentation" This reverts commit c8f8682. * Documentation updates for clarity on Eager mode / Torch scripted models (#439) * Model Zoo updates for Eager Mode / Torchscripted models * Update examples/README.md Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * Apachebench benchmarking - Script / Documentation updates (#438) * Apachebench benchmarking script - Script / Documentation updates * Stop TS after benchmark completion. * Adding system diagram and brief explanation of workings (#413) * Adding system diagram and brief explanation of workings * Adding number of worker recommendations * Adding description of netty configuration params * Update header for worker threads readme section Co-authored-by: mycpuorg <mail@mycpu.org> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Added TODO statement * rebased to master * rebased to master Co-authored-by: dhaniram-kshirsagar <26479924+dhaniram-kshirsagar@users.noreply.github.com> Co-authored-by: dhaniram kshirsagar <dhaniram_kshirsagar@persistent.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: mycpuorg <mail@mycpu.org> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> Co-authored-by: mjpsl <57522192+mjpsl@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-35-42.ec2.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-42-124.ec2.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-67.ec2.internal> Co-authored-by: Vamshidhar Dantu <36211508+vdantu@users.noreply.github.com> Co-authored-by: Vamshidhar Dantu <dantu@amazon.com> Co-authored-by: Brad Heintz <bradheintz@fb.com> Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaron Markham <markhama@amazon.com> Co-authored-by: alexwong <11878166+alexwong@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-12-178.us-west-2.compute.internal> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-0-61.us-west-2.compute.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-7-197.us-west-2.compute.internal> Co-authored-by: Dhanasekar Karuppasamy <dhanak@amazon.com> Co-authored-by: dkasar-aws <57526278+dkasar-aws@users.noreply.github.com> Co-authored-by: Shivam Shriwas <shivamshriwas21@gmail.com> Co-authored-by: Hamid Shojanazeri <hamid.nazeri2010@gmail.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-14-91.us-west-2.compute.internal> Co-authored-by: shivamshriwas <44968110+shivamshriwas@users.noreply.github.com> Co-authored-by: Takayoshi Nishida <takayoshi.nishida@gmail.com> Co-authored-by: alvations <alvations@gmail.com> Co-authored-by: Tzu-Wei Huang <huang.dexter@gmail.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-64-54.ec2.internal> Co-authored-by: Francesco Saverio Zuppichini <zuppif@usi.ch> Co-authored-by: Ubuntu <ubuntu@ip-172-31-80-124.ec2.internal> Co-authored-by: Jessica Lin <jplin@fb.com> Co-authored-by: Mario Cho <hephaex@gmail.com> Co-authored-by: Eli Uriegas <1700823+seemethere@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-78-248.ec2.internal>

…h#433) * made model server port tests independent * Add integration tests for Huggingface transfromers with new handler that support torchscript Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: mycpuorg <mail@mycpu.org> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com>

* fixed resnet152 custom handler to work with GPU * issue template * Moving issue templates to root folder per pull request template * adding feature request template * Minor changes related to sections and names. * Update feature_template.md * Update bug_template.md * doc template * minor changes * Minor changes to upload custom model or handler * issue templates (pytorch#273) * issue template * Moving issue templates to root folder per pull request template * adding feature request template * Minor changes related to sections and names. * Update feature_template.md * Update bug_template.md * doc template * minor changes * Minor changes to upload custom model or handler Co-authored-by: mycpuorg <mail@mycpu.org> * Fixes pytorch#205 - Docker install not finding GPUs (pytorch#262) Invokes docker with --runtime=nvidia for GPU Added the option to specify the GPU / specific GPU ids in start.sh Fixed the documentation for start.sh script Fixed the JDK version in Dockerfile.gpu Co-authored-by: Aaqib <maaquib@gmail.com> * Fix pylint issue (pytorch#163) * pylint fixes and code cleanup * UT fixes, pylint updates per latest version and updated sanity script to run pylint * fixed pylint and wrapt installation issue * freeze pylint version to 2.4.4 due to bug in version 2.5 * removed duplicate statement to install pylint Co-authored-by: Aaqib <maaquib@gmail.com> * Updated exception handling for worker threads shutdown. (pytorch#172) * updated code to now log errors only in case when exception is not genreated by unregister api call * java formatting * fixed illigalstateexception * removed the unrequired handling for illegal monitor state exception Co-authored-by: dhaniram kshirsagar <dhaniram_kshirsagar@persistent.com> Co-authored-by: Aaqib <maaquib@gmail.com> * changes for Homebrew issue 98 (pytorch#126) * changes for Homebrew issue 98 * adding changes for path in benchmark.py * CI Build corrected * benchmark changes corrected path * changed PATH for homebrew * Update Readme Made changes for homebrew installation and instruction for GPU based machine instruction for install_dependencies.sh * Updated Readme.md removed unnecessory lines * corrected indentation corrected indentation for new lines added * Update Readme.md fil added separate commands for CPU and GPU based Co-authored-by: Ubuntu <ubuntu@ip-172-31-35-42.ec2.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-42-124.ec2.internal> Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-67.ec2.internal> Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * added changes to clean model loaded through URL from model store on unregisted * added UTs * Moving the plugins SDK to org.pytorch. Added plugins example. (pytorch#261) * Moved plugins SDK to org.pytorch * Updating the repo for publishing plugins SDK * Enabled bintray maven repo temporarily * Reset tests * Added GPG plugin to pom file to sign the plugins SDK * Update the SDK version * Removed bintray repo * Modify the buildspec to not deploy the SDK Co-authored-by: Vamshidhar Dantu <dantu@amazon.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: mycpuorg <mail@mycpu.org> * Added model zoo (pytorch#299) Co-authored-by: mycpuorg <mail@mycpu.org> Co-authored-by: Brad Heintz <bradheintz@fb.com> * Removed snapshot dependency on model (pytorch#152) * Snapshot is now independent of model * fixed checkstyle issue * fixed formatting * doc review: configuration and custom_service * addressed review comments * fixed incorrect merge * fixed review comments and corrected the error messages * adding remaining fixes for string constants Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaron Markham <markhama@amazon.com> Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> * Regression Tests Suite - AWS Code Build Infra - Initial Commit for pytorch#57 (pytorch#285) * Regression Tests Suite - AWS Code Build Infra - Initial Commit for pytorch#57 * Script to clone Pytorch Repo & Run Post Man Scripts * Basic Tests for Inference & Managment APIs * AWS Code Build buildspec.yml * Refactor postman tests, add empty postman collection for todos * Add a readme for adding tests * Add more management tests * Fix typo * Snapshot Regression tests - pytorch#57 * Fix typo * Comment out snapshot test for now * Weird typos are showing up, try saving everything * Need to manually save every request in postman... * When reregister don't use s3 * More workarounds for rereg * Fix test messages * Remove test for empty reqs * Change get to post * Fix pytest for subprocess - pytorch#57 * Enable pytest - pytorch#57 * Inference API Testcases / Rename Management Testcases pytorch#57 * Wipe Model MARs in Modelstore between tests pytorch#54 * Fix Inference Endpoint Execution pytorch#54 * Fix file paths - pytorch#57 * Fix Initial Workers pytorch#57 * Fix Inference Assertions pytorch#57 * Fix DenseNet Inference pytorch#57 * Fix Densenet Register pytorch#54 * Redirect Logs to File pytorch#54 * Fix Test Execution Log pytorch#54 * README, Build Badge pytorch#57 * Format ReadMe pytorch#57 * Update README - Add Instruction to add new tests / Run tests locally * Update README * Update branch for build Codebuild Badge Co-authored-by: alexwong <11878166+alexwong@users.noreply.github.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: mycpuorg <mail@mycpu.org> Co-authored-by: Ubuntu <ubuntu@ip-172-31-12-178.us-west-2.compute.internal> * Issue 312 install from src (pytorch#322) * documentation: fixed issues in installation and quick start in README.md * documentation: Reverting conda related doc changes from PR#286 Fixes pytorch#297 * remove minus y (pytorch#311) update pip install instructions * Update instructions for installation from source This commit aims to address Issue pytorch#312 Tested on a fresh EC2 P3.2 instance: * Followed instructions under https://github.com/pytorch/serve#install-torchserve-for-development * Started the server manually from command line * Ran a ping query Received a response from server with "Healthy" status Signed-off-by: Manoj Rao <mail@mycpu.org> * add separate files for installation one each for linux and macos * update README and remove relative path from script * Install script - Dont uninstall if not previously installed - pytorch#132 * explicitly build the source code with gradle prior to install * remove upgrade instructions in favor of clean reinstall from source * update instructions to contain the correct script name Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaqib Ansari <maaquib@gmail.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaron Markham <markhama@amazon.com> Co-authored-by: Manoj Rao <mail@mycpu.org> Co-authored-by: Ubuntu <ubuntu@ip-172-31-0-61.us-west-2.compute.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-7-197.us-west-2.compute.internal> * fix for macos script (pytorch#338) * Move Issue Template to .github dir * Updated sanity script to use install_from_src_ubuntu script (pytorch#350) * updated sanity script to use install_from_src_ubuntu script * added java version check * fixed issue with if statement * fixed variable name * fixed repeated cd command * updated artifacts * removed artifacts from buildspec * Corrected HTTP response code for Model version does not exists exception (pytorch#282) * issue 196 patch * Added openapi missing responses. - incorporated review comments * updated method signature after resolving merge conflicts Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Shivam Shriwas <shivamshriwas21@gmail.com> Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Corrected paths in command and added notes (pytorch#354) Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Refactored UT for better reporting (pytorch#159) * refactored UTs * upgraded testng version and updated testcase names * added testng.xml to handle sequential execution of test classes * added waveglow example * Updated readme for waveglow example * fixed failing UTs after merging master * removed unintentionally pushed waveglow example * fixed failing test case * moved set-default to utils and added couple of more add cases for default version * java formatting * corrected set default negative test case * fixed java formatting Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Huggingface Transformers generalization (pytorch#302) * Scripts to seriliaze a Bert model using jit.trace and a handler for Bert model * modified to include mapping of predicitons and decoding received text * move back the original text * moved the index_to_name to bert folder * Transformers/ Bert serving with Torchserve readme * deleting an unused dictionary * just cleaning some of the commented out parts * cleaning up * Cleaning up * uncommenting the logging and cleaning the comments * Download and save the checkpoints, config file and vocab of the BERT from Transformers for demonstration purposes * mapping of predicitons to labels * a custom handler for transformers. * README. * sample of input text. * modifying the descriptions * modifying the descriptions * adding comments * adding commnets * Fromatting modifications * modified comments * added the citations * adding hugginface as the type of transformer model we are referring to * renaming the folder to huggingface_transformers * Removing the bert folder * simple modification * modifed the script to use a setup_config.json for loading and saving models * added the section for transfomer_handler_generalized * the generalized handler * set_up config intialization * edited for transfomer generalization * modified to use just input_ids as inputs for models * modified for question answering * modified to include the number of labels * corrected typos * modifed variable names * edited for clarity * modifed variable names, simplified the initlialize in the handler class, removed the warnning from QA case * modifed values for QA * modifed the variable names * added comments * modified to add clarity * add comment in preprocessing of QA * added warning for the case mode is not selected properly and combined constructor and initlaize fucntion * clean up * clean up * combined constructor and initliaze func * combined constructor and initliaze func * Separated the constructor and initliaze func * Modified to remove the classification part and update with generalized version * resolved the coflict and separated the costructor and initliaze func * removed the handler for classification case as the generalized version handles the case Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: mycpuorg <mail@mycpu.org> * added waveglow example (pytorch#258) * added waveglow example * Updated readme for waveglow example * enhanced example to remove runtime checkpoint download * updated handler to return the audio file content instead of dumping it to tmp dir * minor change to handler * added pipefail * added python snippet to run inference * updated readme with know issue with the example Co-authored-by: Aaqib <maaquib@gmail.com> * made build suite compatible with latest gradle (pytorch#348) Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaqib <maaquib@gmail.com> * moved env section after context * Deleting redundant ISSUE_TEMPLATE at root dir * Fix typo in scripts/install_from_src_macos * Corrected OpenApi description (pytorch#328) * Correct query parameter name (Registration Api)in OpenApi description * removed non existing path parameter Co-authored-by: Aaqib <maaquib@gmail.com> * Regression suite fixes and added test cases (pytorch#344) * added test model zoo test cases updated regression script * Updated Readme * update -y flag for install * Updated docker command as per new imagename and tag generated in stagging branch * Fix sudo usage in regression test script * - addressed review comments - reverted changes related to sudo in regression script. * added https test collection * - updated failed test cases on staging branch - added environment and updated newman cmd with --insecure for https * updated script * updated pytest * update curl check for serve up * Revert "update curl check for serve up" This reverts commit 0047eae. * Revert "update curl check for serve up" This reverts commit 0047eae. - update model zoo - added resnet test case * added workers to resnet model * Update readme to run test suite on desired branch * enhanced inference test cases to driven by datafile * Updated readme Co-authored-by: Dhanasekar Karuppasamy <dhanak@amazon.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Fix typo in mnist readme (pytorch#330) Co-authored-by: Aaqib <maaquib@gmail.com> * Fixed typo in ab installation. (pytorch#327) * documentation: fixed issues in installation and quick start in README.md * documentation: Reverting conda related doc changes from PR#286 Fixes pytorch#297 * remove minus y (pytorch#311) update pip install instructions * Fixed typo in ab installation. Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaqib Ansari <maaquib@gmail.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaron Markham <markhama@amazon.com> * Setup Nightly Build for TorchServe (pytorch#371) * Add Branch Flag for Nightly CodeBuild pytorch#57 * Update ENV Variable for Nightly Build Name * Nigtly Build - Fix FileName for Build Artifacts * Update README for accessing test logs * Whitspace changes * Build new images for build - README Instruction * Fix for intermittent gradle build failure (pytorch#352) * Fix for intermittent null pointer exception * Fixed comments Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> Co-authored-by: Aaqib <maaquib@gmail.com> * documentation: Remove redundant curl options in example (pytorch#336) * documentation: fixed issues in installation and quick start in README.md * documentation: Reverting conda related doc changes from PR#286 Fixes pytorch#297 * remove minus y (pytorch#311) update pip install instructions * fix-doc * fix remaining "-X POST"s Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaqib Ansari <maaquib@gmail.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaron Markham <markhama@amazon.com> * Corrected port number validation range (pytorch#304) * Corrected port number validation range * added UT for issue 291 * added ut dependency file formatting * correct param in config_test_env.properties * removed incorrect flags from test case config files * added negative test cases * removed properties file and changes in test case Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> * Fixes version number reporting for TorchServe (pytorch#360) * Correct version and added vesion.txt for getting version info * fixed description test cases and build issue * Added snapshot test fixes and resolved relative path issue for version.txt * python formatting * added version.txt in MANIFEST.in * addressed review comments * added version.txt for model archiver as well - addressed review comments * -fixed python UT -added ts python UT run in sanity script * removed blank lines from snapshot files Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> * fixes installation script exits without error if no java present on machine (pytorch#376) * fixes installation script exits without error if no java present on machine * added changes for macos script Co-authored-by: Aaqib <maaquib@gmail.com> * Dockerfile enhancements (pytorch#308) * dockerfile now points to latest torchserve on pypi and added separate files for dev * Minor changes * Made build and run commands consistent * Fixing formatting * Consistency issue * Enhanced Dockerfile for binary per pytorch#178 and other misc. * Fixed gpu related issues in start script * Corrected the heading font * Added mar creation steps from container * added buildkit support for dev docker files * Fix for buildkit * Restoring psutil and future as absence leads intermittent hangged build * Fixed review comments for doc * Fixed typo Co-authored-by: Ubuntu <ubuntu@ip-172-31-64-54.ec2.internal> Co-authored-by: Aaqib <maaquib@gmail.com> * Updated configuration.md for missing config properties (pytorch#337) * Updated configuration.md for missing parameters * changes TS_MODEL_SNAPSHOT to MODEL_SNAPSHOT - removes ambiguity - follows naming convention * addressed review comments. * removed debug property * updated configuration.md Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * Adding missing document links to README (pytorch#379) * Maven tests should output only errors (pytorch#326) Co-authored-by: Vamshidhar Dantu <dantu@amazon.com> Co-authored-by: Aaqib <maaquib@gmail.com> * fixed code for build failure in python virtual environment (pytorch#341) * fixed code for build failure in python virtual environment * added exception missed on resolving conflicts Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Vamshidhar Dantu <36211508+vdantu@users.noreply.github.com> * updated object detector handler to use only default cuda device for torchvision version less then 0.6 (pytorch#265) * updated object detector handler to use default cuda device for torchvision version 0.5.0 * updated version check logic * Updated documentation for torchvision version dependency * enhanced sanity script to validate all default handlers * fixed sanity script * added torchtext install step in sanity script * fixed issue related to inference after restart with snapshot. * moved torchtext to codebuild docker files * added more log statements * enhanced run inference function * fixed register model function * added unregister model function * incorporated code review comments * added back inference call for snapshot test * fixed if block Co-authored-by: Aaqib <maaquib@gmail.com> * Adding a barebones ec2 template to run torchserve on ubuntu (pytorch#301) * Adding a barebones ec2 template to run torchserve on ubuntu * Adding VPC with Private subnet * Address review comments #1 * Address review comments pytorch#2 * Rebase for pytorch#365 (pytorch#389) * Updated README.md for pytorch#364 See pytorch#364 for a deep description of the problem * updated README.md * updated README.md Co-authored-by: Francesco Saverio Zuppichini <zuppif@usi.ch> * Document the docker settings for running in production mode (pytorch#388) * Docker Production Options pytorch#234 * Docker Production Options - Typo Fixes / Fit & Finish pytorch#234 * Typos - Docker Docs pytorch#234 * grammar check - docker/README.md Co-authored-by: Aaqib <maaquib@gmail.com> * grammar check - docker/README.md Co-authored-by: Aaqib <maaquib@gmail.com> * grammar check - docker/README.md Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * Huggingface transformers integration test (pytorch#382) * Add Huggingface transformers example integration tests * Add transformers intallation * Fix a missing bracket * move transformers installation to chain installations Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> * Updated docker README.md (pytorch#401) * Update README.md * Fixed cd command * Made source cloning steps explicit!! * Fixed online editing mistake Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> * Fixed response status message for inferencing on model with zero workers (pytorch#305) * Changes for issue_29 * Changed error Messages * changed message for scalling request * changed defaultVersion string * Corrected error message * changed the code for best practice * resolved review comment for version * updated management api doc as per new responses in worker scale api Co-authored-by: Ubuntu <ubuntu@ip-172-31-80-124.ec2.internal> Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> Co-authored-by: Shivam Shriwas <shivamshriwas21@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * made model server port tests independent (pytorch#396) * Added backend profiling and fixed benchmark's install dependency scripts (pytorch#402) * added backend profiling * updated install dependency scripts to make them jmeter version independent * updated readme * Fix benchmarks/README typo and JMeter versioning issue * updated documentation Co-authored-by: Aaqib Ansari <maaquib@gmail.com> * Add sphinx directory and conf.py (pytorch#274) * Add sphinx folder that contains files for docs build * Delete .DS_Store * Delete .DS_Store * Add conf.py in docs/ * documentation: fixed issues in installation and quick start in README.md * documentation: Reverting conda related doc changes from PR#286 Fixes pytorch#297 * remove minus y (pytorch#311) update pip install instructions * Update to the latest version of sphinx * Add recommonmark to extensions Add recommonmark to extensions so sphinx project has markdown support. Configuration details here: https://www.sphinx-doc.org/en/master/usage/markdown.html * Update source suffix to use latest recommonmark syntax Updating to the syntax found here: https://www.sphinx-doc.org/en/master/usage/markdown.html Co-authored-by: Aaron Markham <markhama@amazon.com> * Remove old recommonmark parser syntax * Pin sphinx version to work with latest version of recommonmark Co-authored-by: Aaron Markham <markhama@amazon.com> * Pin version latest version recommonmark Co-authored-by: Aaron Markham <markhama@amazon.com> * Delete theme file generated during build * de-duplicate install package of psutil (pytorch#329) The Dockerfile of GPU install package psutil duplicated. * Remove pytorch-sphinx-theme files * Update to match latest staging_0_1_1 * Update docker directory to match staging_0_1_1 branch * Delete .DS_Store * Add sphinxcontrib.katex and torch Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaqib Ansari <maaquib@gmail.com> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Aaron Markham <markhama@amazon.com> Co-authored-by: Mario Cho <hephaex@gmail.com> Co-authored-by: mycpuorg <mail@mycpu.org> * Image classifier example readme fixes (pytorch#405) * Changed the model name in torch.jit.save to be consistent with docs * Remove extra handler flag and fix the torchservr start * fix the doc * fix the doc Co-authored-by: mycpuorg <mail@mycpu.org> * Porting memory fix changes to TorchServe (pytorch#422) Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Benchmark documentation fix (pytorch#425) * Fix Benchmark Documentation * Documentaion fixes * .circleci: Add initial circleci configuration (pytorch#431) Signed-off-by: Eli Uriegas <eliuriegas@fb.com> * Torchscript support huggingface transfrmers (pytorch#421) * Add support for torchscript * Add support for torchscript * Add key values for torchscript support * Add Torchscript suport * Add Torchscript support * Add Torchscript support in loading the models * keep the example to QA * modify to reflect Torchscript example? * Add torchscript support guides * Add fintued BERT for QA * Add QA artifacts * Add seqenece classification artifacts * Token classification artifacts * Correct typos * Add max length to tokenizer * clean up * enhance the doc * change setup to seq classification * Add detail to torchscript support * set the model for QA * Add torchscript bert seq classification example * Add bert seq classification example * Modify the packaging model command * Modify torchscript packaging command * clean up * clean up * correct spacing between list items Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Integration test huggingface transfromers torchscript support (pytorch#433) * made model server port tests independent * Add integration tests for Huggingface transfromers with new handler that support torchscript Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: mycpuorg <mail@mycpu.org> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: dhanainme <60679183+dhanainme@users.noreply.github.com> * Apachebench benchmark - cleanup script / documentation * Revert "Apachebench benchmark - cleanup script / documentation" This reverts commit c8f8682. * Documentation updates for clarity on Eager mode / Torch scripted models (pytorch#439) * Model Zoo updates for Eager Mode / Torchscripted models * Update examples/README.md Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: Aaqib <maaquib@gmail.com> * Apachebench benchmarking - Script / Documentation updates (pytorch#438) * Apachebench benchmarking script - Script / Documentation updates * Stop TS after benchmark completion. Co-authored-by: harshbafna <harsh_bafna@persistent.co.in> Co-authored-by: dhaniram-kshirsagar <26479924+dhaniram-kshirsagar@users.noreply.github.com> Co-authored-by: dhaniram kshirsagar <dhaniram_kshirsagar@persistent.com> Co-authored-by: Aaqib <maaquib@gmail.com> Co-authored-by: mycpuorg <mail@mycpu.org> Co-authored-by: Harsh Bafna <harshbafna619@gmail.com> Co-authored-by: mjpsl <57522192+mjpsl@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-35-42.ec2.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-42-124.ec2.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-67.ec2.internal> Co-authored-by: Vamshidhar Dantu <36211508+vdantu@users.noreply.github.com> Co-authored-by: Vamshidhar Dantu <dantu@amazon.com> Co-authored-by: Brad Heintz <bradheintz@fb.com> Co-authored-by: eslesar-aws <eslesar@amazon.com> Co-authored-by: Aaron Markham <markhama@amazon.com> Co-authored-by: alexwong <11878166+alexwong@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-12-178.us-west-2.compute.internal> Co-authored-by: Geeta Chauhan <4461127+chauhang@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-0-61.us-west-2.compute.internal> Co-authored-by: Ubuntu <ubuntu@ip-172-31-7-197.us-west-2.compute.internal> Co-authored-by: dkasar-aws <57526278+dkasar-aws@users.noreply.github.com> Co-authored-by: Shivam Shriwas <shivamshriwas21@gmail.com> Co-authored-by: Hamid Shojanazeri <hamid.nazeri2010@gmail.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-14-91.us-west-2.compute.internal> Co-authored-by: shivamshriwas <44968110+shivamshriwas@users.noreply.github.com> Co-authored-by: Takayoshi Nishida <takayoshi.nishida@gmail.com> Co-authored-by: alvations <alvations@gmail.com> Co-authored-by: Tzu-Wei Huang <huang.dexter@gmail.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-64-54.ec2.internal> Co-authored-by: Francesco Saverio Zuppichini <zuppif@usi.ch> Co-authored-by: Ubuntu <ubuntu@ip-172-31-80-124.ec2.internal> Co-authored-by: Jessica Lin <jplin@fb.com> Co-authored-by: Mario Cho <hephaex@gmail.com> Co-authored-by: Eli Uriegas <1700823+seemethere@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-172-31-78-248.ec2.internal>

harshbafna and others added 3 commits May 25, 2020 15:56

made model server port tests independent

95c2711

Merge branch 'staging_0_1_1' into issue_391

acc7e14

Add integration tests for Huggingface transfromers with new handler t…

33d14d8

…hat support torchscript

Merge branch 'staging_0_1_1' into integration_test_Huggingface_transf…

cc402ce

…romers_torchscriptSupport

chauhang approved these changes Jun 5, 2020

View reviewed changes

Merge branch 'staging_0_1_1' into integration_test_Huggingface_transf…

4f758cb

…romers_torchscriptSupport

Merge branch 'staging_0_1_1' into integration_test_Huggingface_transf…

705062d

…romers_torchscriptSupport

dhanainme merged commit 8b67e7a into pytorch:staging_0_1_1 Jun 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integration test huggingface transfromers torchscript support #433

Integration test huggingface transfromers torchscript support #433

HamidShojanazeri commented Jun 5, 2020

sagemaker-neo-ci-bot commented Jun 5, 2020

sagemaker-neo-ci-bot commented Jun 5, 2020

sagemaker-neo-ci-bot commented Jun 5, 2020

sagemaker-neo-ci-bot commented Jun 5, 2020

sagemaker-neo-ci-bot commented Jun 5, 2020

sagemaker-neo-ci-bot commented Jun 5, 2020

sagemaker-neo-ci-bot commented Jun 5, 2020

sagemaker-neo-ci-bot commented Jun 5, 2020

sagemaker-neo-ci-bot commented Jun 5, 2020

sagemaker-neo-ci-bot commented Jun 5, 2020

Integration test huggingface transfromers torchscript support #433

Integration test huggingface transfromers torchscript support #433

Conversation

HamidShojanazeri commented Jun 5, 2020

Description

Feature/Issue validation/testing.

Checklist:

sagemaker-neo-ci-bot commented Jun 5, 2020

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Jun 5, 2020

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Jun 5, 2020

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Jun 5, 2020

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Jun 5, 2020

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Jun 5, 2020

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Jun 5, 2020

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Jun 5, 2020

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Jun 5, 2020

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Jun 5, 2020

AWS CodeBuild CI Report