Merge branch 'master' into issue_2090

.github/workflows/regression_tests_gpu.yml

@@ -5,7 +5,7 @@ on: workflow_dispatch
 jobs:
   regression-gpu:
     # creates workflows for CUDA 11.6 & CUDA 11.7 on ubuntu
-    runs-on: [self-hosted, ci-gpu]
+    runs-on: [self-hosted, regression-test-gpu]
     strategy:
       fail-fast: false
       matrix:

README.md

@@ -76,7 +76,9 @@ Refer to [torchserve docker](docker/README.md) for details.
 * [Model parallel inference](examples/Huggingface_Transformers#model-parallelism)
 * [MultiModal models with MMF](https://github.com/pytorch/serve/tree/master/examples/MMF-activity-recognition) combining text, audio and video
 * [Dual Neural Machine Translation](examples/Workflows/nmt_transformers_pipeline) for a complex workflow DAG
-
+* [TorchServe Integrations](examples/README.md#torchserve-integrations)
+* [TorchServe Internals](examples/README.md#torchserve-internals)
+* [TorchServe UseCases](examples/README.md#usecases)
 
 For [more examples](examples/README.md)
 

docs/README.md

@@ -33,9 +33,12 @@ TorchServe is a performant, flexible and easy to use tool for serving PyTorch ea
 
 * [HuggingFace Language Model](https://github.com/pytorch/serve/blob/master/examples/Huggingface_Transformers/Transformer_handler_generalized.py) - This handler takes an input sentence and can return sequence classifications, token classifications or Q&A answers
 * [Multi Modal Framework](https://github.com/pytorch/serve/blob/master/examples/MMF-activity-recognition/handler.py) - Build and deploy a classifier that combines text, audio and video input data
-* [Dual Translation Workflow](https://github.com/pytorch/serve/tree/master/examples/Workflows/nmt_transformers_pipeline) - 
+* [Dual Translation Workflow](https://github.com/pytorch/serve/tree/master/examples/Workflows/nmt_transformers_pipeline) -
 * [Model Zoo](model_zoo.md) - List of pre-trained model archives ready to be served for inference with TorchServe.
 * [Examples](https://github.com/pytorch/serve/tree/master/examples) - Many examples of how to package and deploy models with TorchServe
+     - [TorchServe Internals](../examples/README.md#torchserve-internals)
+     - [TorchServe Integrations](../examples/README.md#torchserve-integrations)
+     - [TorchServe UseCases](../examples/README.md#usecases)
 * [Workflow Examples](https://github.com/pytorch/serve/tree/master/examples/Workflows) - Examples of how to compose models in a workflow with TorchServe
 
 ## Advanced Features

docs/grpc_api.md

@@ -28,7 +28,7 @@ Run following commands to Register, run inference and unregister, densenet161 mo
  - [Install TorchServe](../README.md)
 
  - Clone serve repo to run this example
- 
+
 ```bash
 git clone https://github.com/pytorch/serve
 cd serve
@@ -43,24 +43,24 @@ pip install -U grpcio protobuf grpcio-tools
  - Start torchServe
 
 ```bash
-mkdir model_store
-torchserve --start 
+mkdir models
+torchserve --start --model-store models/
 ```
 
  - Generate python gRPC client stub using the proto files
- 
+
 ```bash
 python -m grpc_tools.protoc --proto_path=frontend/server/src/main/resources/proto/ --python_out=ts_scripts --grpc_python_out=ts_scripts frontend/server/src/main/resources/proto/inference.proto frontend/server/src/main/resources/proto/management.proto
 ```
 
  - Register densenet161 model
- 
+
 ```bash
 python ts_scripts/torchserve_grpc_client.py register densenet161
 ```
 
- - Run inference using 
- 
+ - Run inference using
+
 ```bash
 python ts_scripts/torchserve_grpc_client.py infer densenet161 examples/image_classifier/kitten.jpg
 ```

examples/README.md

@@ -1,22 +1,103 @@
-# Contents of this Document
+# Examples showcasing TorchServe Features and Integrations
+
+## TorchServe Internals
+
 * [Creating mar file for an eager mode model](#creating-mar-file-for-eager-mode-model)
 * [Creating mar file for torchscript mode model](#creating-mar-file-for-torchscript-mode-model)
-* [Serving torchvision image classification models](#serving-image-classification-models)
 * [Serving custom model with custom service handler](#serving-custom-model-with-custom-service-handler)
-* [Serving text classification model](#serving-text-classification-model)
-* [Serving text classification model with scriptable tokenizer](#serving-text-classification-model-with-scriptable-tokenizer)
-* [Serving object detection model](#serving-object-detection-model)
-* [Serving image segmentation model](#serving-image-segmentation-model)
-* [Serving huggingface transformers model](#serving-huggingface-transformers)
-* [Serving image generator model](#example-to-serve-gan-model)
-* [Serving machine translation model](#serving-neural-machine-translation)
-* [Serving waveglow text to speech synthesizer model](#serving-waveglow-text-to-speech-synthesizer)
-* [Serving multi modal framework model](#serving-multi-modal-model)
-* [Serving Image Classification Workflow](#serving-image-classification-workflow)
-* [Serving Neural Machine Translation Workflow](#serving-neural-machine-translation-workflow)
-* [Serving Torchrec DLRM (Recommender Model)](#serving-torchrec-dlrm-recommender-model)
-* [Serving Image Classifier model for on-premise near real-time video](#serving-image-classifier-model-for-on-premise-near-real-time-video)
-* [Serving Image Classifier model and loading image data using torchdata (datapipes)](#serving-image-classifier-model-with-torchdata-datapipes)
+* [Creating a Workflow](Workflows/dog_breed_classification)
+* [Custom Metrics](custom_metrics)
+* [Dynamic Batch Processing](image_classifier/resnet_152_batch)
+* [Dynamic Batched Async Requests](image_classifier/near_real_time_video)
+
+## TorchServe Integrations
+
+### Kubernetes <img src="images/k8s.png" width="30"  title="K8s" style="float:right padding:10px" />
+
+* [Serving HuggingFace faster transformers model in K8s](../kubernetes/examples/FasterTransformer_HuggingFace_Bert.md)
+
+### KServe <img src="images/kserve.png" width="40"  title="KServe" style="float:right padding:10px" />
+
+* [Serving HuggingFace BERT model using KServe](https://github.com/kserve/kserve/blob/master/docs/samples/v1beta1/custom/torchserve/bert-sample/hugging-face-bert-sample.md)
+
+### Hugging Face <img src="images/huggingface_logo-noborder.svg" width="30" height="30" title="Hugging Face" style="float:right padding:10px" />
+
+* [Serving HuggingFace transformers model](Huggingface_Transformers)
+
+### MLFlow <img src="images/mlflow.png" width="50"  title="MLFlow" style="float:right padding:20px" />
+
+* [Deploy models using `mlflow-torchserve` plugin](https://github.com/mlflow/mlflow-torchserve/tree/master/examples)
+
+### Captum <img src="images/captum_logo.svg" width="50" title="Captum"  />
+
+* [Model Explainability with Captum](captum)
+
+### ONNX <img src="images/onnx.png" width="80" title="ONNX" style="float:right padding:20px" />
+
+* [Example for ONNX Integration](../test/pytest/test_onnx.py)
+
+### TensorRT
+
+* [Support for TensorRT optimizations](../docs/performance_guide.md#tensorrt-and-nvfuser-support)
+
+### Microsoft DeepSpeed-MII <img src="images/mii-white.svg" width="80" title="DeepSpeed MII" style="float:top" />
+
+* [HuggingFace Stable Diffusion Model with Microsoft DeepSpeed-MII](deepspeed_mii)
+
+### Prometheus and mtail <img src="images/prometheus-logo.svg" width="30" title="Prometheus" style="float:top" />
+
+* [Custom Metrics with mtail and Prometheus](custom_metrics) <img src="images/mtail.png" width="30" title="mtail" style="float:top" />
+
+### Intel® Extension for PyTorch
+* [ Boost Performance on Intel Hardware](intel_extension_for_pytorch)
+
+### TorchRec DLRM
+<img src="images/dlrm.png" width="400" title="DLRM"/>
+
+* [Serving Torchrec DLRM (Recommender Model)](torchrec_dlrm)
+
+### TorchData
+* [Serving Image Classifier model and loading image data using TorchData (datapipes)](image_classifier/mnist/torchdata)
+
+### PyTorch 2.0
+* [PyTorch 2.0 Integration](pt2)
+
+### Stable Diffusion <img src="images/huggingface_logo-noborder.svg" width="30" height="30" title="Hugging Face" style="float:right padding:10px" />
+* [Stable Diffusion using HuggingFace Diffusers](diffusers)
+
+### HuggingFace Large Models <img src="images/huggingface_logo-noborder.svg" width="30" height="30" title="Hugging Face" style="float:right padding:10px" />
+* [HuggingFace Large Models with constrained resources](Huggingface_Largemodels)
+
+## UseCases
+
+### Vision
+#### Image Classification
+* [Serving torchvision image classification models](image_classifier)
+* [Serving Image Classifier model for on-premise near real-time video](image_classifier/near_real_time_video)
+
+#### Object Detection
+* [Serving object detection model](object_detector)
+* [Serving image segmentation model](image_segmenter)
+
+#### GAN
+* [Serving image generator model](dcgan_fashiongen)
+
+### Text
+#### Neural Machine Translation
+<img src="images/seq2seq.png" width="400" title="SEQ2SEQ"/>
+
+* [Serving machine translation model](nmt_transformer)
+* [Serving Neural Machine Translation Workflow](Workflows/nmt_transformers_pipeline)
+
+#### Text Classification
+* [Serving text classification model](text_classification)
+* [Serving text classification model with scriptable tokenizer](text_classification_with_scriptable_tokenizer)
+
+#### Text to Speech
+* [Serving waveglow text to speech synthesizer model](speech2text_wav2vec2)
+
+### MultiModal
+* [Serving multi modal framework model](MMF-activity-recognition)
 
 # TorchServe Examples
 
@@ -95,6 +176,12 @@ The following example demonstrates how to create and serve a pretrained transfor
 
 * [Hugging Face Transformers](Huggingface_Transformers)
 
+## Captum Integration
+
+The following example demonstrates TorchServe's integration with Captum, an open source, extensible library for model interpretability built on PyTorch
+
+* [Captum](captum)
+
 ## Example to serve GAN model
 
 The following example demonstrates how to create and serve a pretrained DCGAN model from [facebookresearch/pytorch_GAN_zoo](https://github.com/facebookresearch/pytorch_GAN_zoo)
@@ -142,8 +229,8 @@ The following example demonstrates how to serve an image classification model wi
 
 * [Near Real-Time Video Batched Image Classification](image_classifier/near_real_time_video)
 
-## Serving Image Classifier Model with torchdata datapipes
+## Serving Image Classifier Model with TorchData datapipes
 
-The following example demonstrates how to integrate torchdata with torchserve
+The following example demonstrates how to integrate TorchData with torchserve
 
 * [Torchdata integration with torchserve an image classification example](image_classifier/mnist/torchdata)