From c911150cbca81fe5f91076a488c40d91239c77ae Mon Sep 17 00:00:00 2001 From: Emma Liu Date: Fri, 21 Jun 2024 16:05:35 -0700 Subject: [PATCH 1/2] Update image_classifier/densenet-161 to include torch.compile --- .../image_classifier/densenet_161/README.md | 34 +++++++++++++++++-- .../densenet_161/model-config.yaml | 5 +++ 2 files changed, 37 insertions(+), 2 deletions(-) create mode 100644 examples/image_classifier/densenet_161/model-config.yaml diff --git a/examples/image_classifier/densenet_161/README.md b/examples/image_classifier/densenet_161/README.md index 758c072722..d9f0f4646a 100644 --- a/examples/image_classifier/densenet_161/README.md +++ b/examples/image_classifier/densenet_161/README.md @@ -35,8 +35,8 @@ model.eval() example_input = torch.rand(1, 3, 224, 224) traced_script_module = torch.jit.trace(model, example_input) traced_script_module.save("densenet161.pt") -``` - +``` + * Use following commands to register Densenet161 torchscript model on TorchServe and run image prediction ```bash @@ -46,3 +46,33 @@ mv densenet161_ts.mar model_store/ torchserve --start --model-store model_store --models densenet161=densenet161_ts.mar curl http://127.0.0.1:8080/predictions/densenet161 -T examples/image_classifier/kitten.jpg ``` + +#### TorchServe inference with torch.compile of densenet161 model +This example shows how to take eager model of `densenet161`, configure TorchServe to use `torch.compile` and run inference using `torch.compile` + +Change directory to the examples directory +`cd examples/image_classifier/densenet_161` + +`torch.compile` supports a variety of config and the performance you get can vary based on the config. You can find the various options [here](https://pytorch.org/docs/stable/generated/torch.compile.html). + +Sample command to start torchserve with torch.compile: + +```bash +wget https://download.pytorch.org/models/densenet161-8d451a50.pth +mkdir model_store +torch-model-archiver --model-name densenet161 --version 1.0 --model-file model.py --serialized-file densenet161-8d451a50.pth --export-path model_store --extra-files ../../image_classifier/index_to_name.json --handler image_classifier --config-file model-config.yaml -f +torchserve --start --ncs --model-store model_store --models densenet161.mar +curl http://127.0.0.1:8080/predictions/densenet161 -T ../../image_classifier/kitten.jpg +``` + +produces the output + +``` +{ + "tabby": 0.4664836823940277, + "tiger_cat": 0.4645617604255676, + "Egyptian_cat": 0.06619937717914581, + "lynx": 0.0012969186063855886, + "plastic_bag": 0.00022856894065625966 +} +``` diff --git a/examples/image_classifier/densenet_161/model-config.yaml b/examples/image_classifier/densenet_161/model-config.yaml new file mode 100644 index 0000000000..b362781ef4 --- /dev/null +++ b/examples/image_classifier/densenet_161/model-config.yaml @@ -0,0 +1,5 @@ +pt2: + compile: + enable: True + backend: inductor + mode: reduce-overhead From a762175e9a96cc73dc5362b8862546a1bc9ca5b7 Mon Sep 17 00:00:00 2001 From: Emma Liu Date: Fri, 21 Jun 2024 16:24:40 -0700 Subject: [PATCH 2/2] minor --- .../image_classifier/densenet_161/README.md | 61 ++++++++++--------- 1 file changed, 31 insertions(+), 30 deletions(-) diff --git a/examples/image_classifier/densenet_161/README.md b/examples/image_classifier/densenet_161/README.md index d9f0f4646a..54c805970a 100644 --- a/examples/image_classifier/densenet_161/README.md +++ b/examples/image_classifier/densenet_161/README.md @@ -1,3 +1,34 @@ +#### TorchServe inference with torch.compile of densenet161 model +This example shows how to take eager model of `densenet161`, configure TorchServe to use `torch.compile` and run inference using `torch.compile` + +Change directory to the examples directory +`cd examples/image_classifier/densenet_161` + +`torch.compile` supports a variety of config and the performance you get can vary based on the config. You can find the various options [here](https://pytorch.org/docs/stable/generated/torch.compile.html). + +Sample command to start torchserve with torch.compile: + +```bash +wget https://download.pytorch.org/models/densenet161-8d451a50.pth +mkdir model_store +torch-model-archiver --model-name densenet161 --version 1.0 --model-file model.py --serialized-file densenet161-8d451a50.pth --export-path model_store --extra-files ../../image_classifier/index_to_name.json --handler image_classifier --config-file model-config.yaml -f +torchserve --start --ncs --model-store model_store --models densenet161.mar +curl http://127.0.0.1:8080/predictions/densenet161 -T ../../image_classifier/kitten.jpg +``` + +produces the output + +``` +{ + "tabby": 0.4664836823940277, + "tiger_cat": 0.4645617604255676, + "Egyptian_cat": 0.06619937717914581, + "lynx": 0.0012969186063855886, + "plastic_bag": 0.00022856894065625966 +} +``` + + #### Sample commands to create a densenet eager mode model archive, register it on TorchServe and run image prediction Run the commands given in following steps from the parent directory of the root of the repository. For example, if you cloned the repository into /home/my_path/serve, run the steps from /home/my_path/serve @@ -46,33 +77,3 @@ mv densenet161_ts.mar model_store/ torchserve --start --model-store model_store --models densenet161=densenet161_ts.mar curl http://127.0.0.1:8080/predictions/densenet161 -T examples/image_classifier/kitten.jpg ``` - -#### TorchServe inference with torch.compile of densenet161 model -This example shows how to take eager model of `densenet161`, configure TorchServe to use `torch.compile` and run inference using `torch.compile` - -Change directory to the examples directory -`cd examples/image_classifier/densenet_161` - -`torch.compile` supports a variety of config and the performance you get can vary based on the config. You can find the various options [here](https://pytorch.org/docs/stable/generated/torch.compile.html). - -Sample command to start torchserve with torch.compile: - -```bash -wget https://download.pytorch.org/models/densenet161-8d451a50.pth -mkdir model_store -torch-model-archiver --model-name densenet161 --version 1.0 --model-file model.py --serialized-file densenet161-8d451a50.pth --export-path model_store --extra-files ../../image_classifier/index_to_name.json --handler image_classifier --config-file model-config.yaml -f -torchserve --start --ncs --model-store model_store --models densenet161.mar -curl http://127.0.0.1:8080/predictions/densenet161 -T ../../image_classifier/kitten.jpg -``` - -produces the output - -``` -{ - "tabby": 0.4664836823940277, - "tiger_cat": 0.4645617604255676, - "Egyptian_cat": 0.06619937717914581, - "lynx": 0.0012969186063855886, - "plastic_bag": 0.00022856894065625966 -} -```