torch compile config standardization update #3166

agunapal · 2024-05-30T17:38:58Z

Description

This PR standardizes how TorchServe would be supporting any PyTorch 2.x APIs config

Define the following structure for a PyTorch 2.x feature config in model-config.yaml

pt2:
  <API name>:
    enable: True
    option1:  value1
    optionN:  valueN

or

pt2:
  <API name>:
    <feature 1>:
      enable: True
      option1:  value1
      optionN:  valueN

For torch.compile, we would specify the following options

pt2:
  compile:
    enable: True
    backend: "inductor"
    mode: "max-autotune"

This PR also fixes the issues with torch compile pytests

Fixes #(
issue1
issue2
issue3
)

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
New feature (non-breaking change which adds functionality)
This change requires a documentation update

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

pytest -v test_torch_compile.py
================================================================================== test session starts ===================================================================================
platform linux -- Python 3.10.14, pytest-7.3.1, pluggy-1.0.0 -- /home/ubuntu/anaconda3/envs/torchserve/bin/python
cachedir: .pytest_cache
rootdir: /home/ubuntu/serve
plugins: cov-4.1.0, mock-3.14.0, timeout-2.3.1
collected 8 items                                                                                                                                                                        

test_torch_compile.py::TestTorchCompile::test_archive_model_artifacts PASSED                                                                                                       [ 12%]
test_torch_compile.py::TestTorchCompile::test_start_torchserve PASSED                                                                                                              [ 25%]
test_torch_compile.py::TestTorchCompile::test_server_status PASSED                                                                                                                 [ 37%]
test_torch_compile.py::TestTorchCompile::test_registered_model PASSED                                                                                                              [ 50%]
test_torch_compile.py::TestTorchCompile::test_serve_inference SKIPPED (Test failing on regression runner)                                                                          [ 62%]
test_torch_compile.py::TestTorchCompile::test_compile_inference_enable_true_default PASSED                                                                                         [ 75%]
test_torch_compile.py::TestTorchCompile::test_compile_inference_enable_true PASSED                                                                                                 [ 87%]
test_torch_compile.py::TestTorchCompile::test_compile_inference_enable_false PASSED                                                                                                [100%]

==================================================================================== warnings summary ====================================================================================
test_torch_compile.py:12
  /home/ubuntu/serve/test/pytest/test_torch_compile.py:12: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging

../../../anaconda3/envs/torchserve/lib/python3.10/site-packages/pkg_resources/__init__.py:2832
../../../anaconda3/envs/torchserve/lib/python3.10/site-packages/pkg_resources/__init__.py:2832
  /home/ubuntu/anaconda3/envs/torchserve/lib/python3.10/site-packages/pkg_resources/__init__.py:2832: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================================= 7 passed, 1 skipped, 3 warnings in 35.29s ========================================================================

Checklist:

Did you have fun?
Have you added tests that prove your fix is effective or that this feature works?
Has code been commented, particularly in hard-to-understand areas?
Have you made corresponding changes to the documentation?

mreso

LGTM, just left some nits

mreso · 2024-06-10T18:02:10Z

examples/image_classifier/resnet_18/README.md

@@ -23,7 +23,7 @@ Ex:  `cd  examples/image_classifier/resnet_18`
 In this example , we use the following config

 ```
-echo "pt2 : {backend: inductor, mode: reduce-overhead}" > model-config.yaml
+echo "pt2:\n  compile:\n    enable: True\n    backend: inductor\n    mode: reduce-overhead" > model-config.yaml


If you just use multiple lines people will be able to visualize the new format much better:

echo "pt2: compile: enable: True backend: inductor mode: reduce-overhead" > model-config.yaml

Makes sense. Done

mreso · 2024-06-10T18:03:20Z

examples/pt2/torch_compile/README.md

@@ -19,7 +19,7 @@ Ex:  `cd  examples/pt2/torch_compile`
 In this example , we use the following config

 ```
-echo "pt2 : {backend: inductor, mode: reduce-overhead}" > model-config.yaml
+echo "pt2:\n  compile:\n    enable: True" > model-config.yaml


See above regarding multiline echo.

Makes sense. Done

mreso · 2024-06-10T18:03:52Z

examples/pt2/torch_compile/README.md

@@ -76,7 +76,7 @@ After a few iterations of warmup, we see the following
 #### Measure inference time with `torch.compile`

 ```
-echo "pt2: {backend: inductor, mode: reduce-overhead}" > model-config.yaml && \
+echo "pt2:\n  compile:\n    enable: True\n    backend: inductor\n    mode: reduce-overhead" > model-config.yaml && \


Makes sense. Done

mreso · 2024-06-10T18:15:36Z

test/pytest/test_torch_compile.py

@@ -146,3 +176,90 @@ def test_serve_inference(self):
                "Compiled model with backend inductor, mode reduce-overhead"
                in model_log
            )
+
+    def test_compile_inference_enable_true_default(self, chdir_example):


You should parametrize these tests to make it more obvious what is the difference between them. See

serve/test/pytest/test_example_gpt_fast.py

Line 98 in 36049cb

@pytest.mark.parametrize(("compile"), ("false", "true"))

for an example

This is cool. Done

…orch/serve into feature/torch_compile_config

agunapal and others added 17 commits May 30, 2024 17:35

torch.compile config update

445cfc8

torch.compile config update

529afb2

yaml test files

42c003f

yaml test files

ffae834

Fixed regression failure

85c7e10

Fixed regression failure

05d87f8

Fixed regression failure

dc8bd90

Workaround for regression failure

302cede

Workaround for regression failure

407e55f

Workaround for regression failure

c632790

skipping torchtext test

88bd261

Update test_example_torch_compile.py

867234d

Update test_torch_compile.py

842c6b2

Rename toy_model.py to model.py

ca640a2

Update test_torch_compile.py

3fc45c7

Update test_torch_compile.py

64d9c55

lint

7a0a31a

agunapal marked this pull request as ready for review June 3, 2024 16:59

agunapal requested a review from mreso June 3, 2024 16:59

agunapal added this to the v0.12.0 milestone Jun 3, 2024

Merge branch 'master' into feature/torch_compile_config

e7d6bf5

agunapal mentioned this pull request Jun 10, 2024

Add support for hpu_backend and Resnet50 compile example #3182

Merged

5 tasks

mreso approved these changes Jun 10, 2024

View reviewed changes

agunapal and others added 4 commits June 11, 2024 17:36

:Addressed review comments

ac8ae47

Merge branch 'master' into feature/torch_compile_config

bd2f012

Addressed review comments

e3d80e2

Merge branch 'feature/torch_compile_config' of https://github.com/pyt…

3613337

…orch/serve into feature/torch_compile_config

agunapal enabled auto-merge June 11, 2024 18:14

agunapal added this pull request to the merge queue Jun 11, 2024

Merged via the queue into master with commit d29059f Jun 11, 2024
11 of 12 checks passed

agunapal deleted the feature/torch_compile_config branch June 11, 2024 20:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

torch compile config standardization update #3166

torch compile config standardization update #3166

agunapal commented May 30, 2024 •

edited

Loading

mreso left a comment

mreso Jun 10, 2024

agunapal Jun 11, 2024

mreso Jun 10, 2024

agunapal Jun 11, 2024

mreso Jun 10, 2024

agunapal Jun 11, 2024

mreso Jun 10, 2024

agunapal Jun 11, 2024

torch compile config standardization update #3166

torch compile config standardization update #3166

Conversation

agunapal commented May 30, 2024 • edited Loading

Description

Type of change

Feature/Issue validation/testing

Checklist:

mreso left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agunapal commented May 30, 2024 •

edited

Loading