Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AOTInductor Resnet CPP example #2944

Merged
merged 18 commits into from
Feb 22, 2024
Merged

AOTInductor Resnet CPP example #2944

merged 18 commits into from
Feb 22, 2024

Conversation

lxning
Copy link
Collaborator

@lxning lxning commented Feb 12, 2024 •
edited
Loading

Description

Please read our CONTRIBUTING.md prior to creating your first pull request.

Please include a summary of the feature or issue being fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #(issue)

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

cd serve/cpp
./build.sh

Running main() from /home/ubuntu/serve.resnet/cpp/_build/_deps/googletest-src/googletest/src/gtest_main.cc
[==========] Running 48 tests from 11 test suites.
[----------] Global test environment set-up.
[----------] 1 test from BackendIntegTest
[ RUN      ] BackendIntegTest.TestOTFProtocolAndHandler
I0217 05:32:09.790347 138546 log_metric.cc:92] [METRICS]HandlerTime.Milliseconds:60.133028|#ModelName:mnist_scripted_v2,Level:Model|#hostname:ip-172-31-35-108,1708147929,reqi
I0217 05:32:09.790414 138546 log_metric.cc:92] [METRICS]PredictionTime.Milliseconds:60.133028|#ModelName:mnist_scripted_v2,Level:Model|#hostname:ip-172-31-35-108,1708147929,reqi
[       OK ] BackendIntegTest.TestOTFProtocolAndHandler (84 ms)
[----------] 1 test from BackendIntegTest (84 ms total)

[----------] 8 tests from OTFMessageTest
[ RUN      ] OTFMessageTest.TestRetieveCmd
[       OK ] OTFMessageTest.TestRetieveCmd (0 ms)
[ RUN      ] OTFMessageTest.TestEncodeLoadModelResponse
[       OK ] OTFMessageTest.TestEncodeLoadModelResponse (0 ms)
[ RUN      ] OTFMessageTest.TestUTF8EncodeLoadModelResponse
[       OK ] OTFMessageTest.TestUTF8EncodeLoadModelResponse (0 ms)
[ RUN      ] OTFMessageTest.TestRetrieveMsgLoadGpu
[       OK ] OTFMessageTest.TestRetrieveMsgLoadGpu (0 ms)
[ RUN      ] OTFMessageTest.TestRetrieveMsgLoadNoGpu
[       OK ] OTFMessageTest.TestRetrieveMsgLoadNoGpu (0 ms)
[ RUN      ] OTFMessageTest.TestEncodeSuccessInferenceResponse
[       OK ] OTFMessageTest.TestEncodeSuccessInferenceResponse (0 ms)
[ RUN      ] OTFMessageTest.TestEncodeFailureInferenceResponse
E0217 05:32:09.792281 138546 otf_message_test.cc:157] result_size: 120
[       OK ] OTFMessageTest.TestEncodeFailureInferenceResponse (0 ms)
[ RUN      ] OTFMessageTest.TestRetrieveInferenceMsg
[       OK ] OTFMessageTest.TestRetrieveInferenceMsg (0 ms)
[----------] 8 tests from OTFMessageTest (0 ms total)

[----------] 10 tests from ModelPredictTest
[ RUN      ] ModelPredictTest.TestLoadPredictBabyLlamaHandler
Total number of tokens generated: 332
Achieved tok per sec: 189.174
Generated String:  Hello my name is
The little girl, who was three years old, was playing in the garden. She saw a big, red tomato and wanted to pick it. She reached out her hand and grabbed it.
Suddenly, a voice said, "Hey! That's my tomato!"
The little girl looked up and saw a big, angry bird. She was scared and started to cry.
The bird said, "That tomato is mine! I'm going to eat it!"
The little girl was very scared and ran away. She was so sad that she didn't get to eat the tomato.
The bird flew away and the little girl was left alone in the garden. She was very sad and cried all the way home.
<s>

Generated String:  Hello my name is
The little girl, who was three years old, was playing in the garden. She saw a big, red tomato and wanted to pick it. She reached out her hand and grabbed it.
Suddenly, a voice said, "Hey! That's my tomato!"
The little girl looked up and saw a big, angry bird. She was scared and started to cry.
The bird said, "That tomato is mine! I'm going to eat it!"
The little girl was very scared and ran away. She was so sad that she didn't get to eat the tomato.
The bird flew away and the little girl was left alone in the garden. She was very sad and cried all the way home.
<s>

I0217 05:32:11.569762 138546 log_metric.cc:92] [METRICS]HandlerTime.Milliseconds:1772.49196|#ModelName:babyllama,Level:Model|#hostname:ip-172-31-35-108,1708147931,llm_ts_0,llm_ts_1
I0217 05:32:11.569796 138546 log_metric.cc:92] [METRICS]PredictionTime.Milliseconds:1772.49196|#ModelName:babyllama,Level:Model|#hostname:ip-172-31-35-108,1708147931,llm_ts_0,llm_ts_1
[       OK ] ModelPredictTest.TestLoadPredictBabyLlamaHandler (1780 ms)
[ RUN      ] ModelPredictTest.TestLoadPredictAotInductorLlamaHandler
I0217 05:32:12.247032 138546 log_metric.cc:92] [METRICS]HandlerTime.Milliseconds:669.318053|#ModelName:llama,Level:Model|#hostname:ip-172-31-35-108,1708147932,llm_ts_0,llm_ts_1
I0217 05:32:12.247074 138546 log_metric.cc:92] [METRICS]PredictionTime.Milliseconds:669.318053|#ModelName:llama,Level:Model|#hostname:ip-172-31-35-108,1708147932,llm_ts_0,llm_ts_1
[       OK ] ModelPredictTest.TestLoadPredictAotInductorLlamaHandler (676 ms)
[ RUN      ] ModelPredictTest.TestLoadPredictLlamaCppHandler
/home/ubuntu/serve.resnet/cpp/test/examples/examples_test.cc:53: Skipped
Skipping TestLoadPredictLlamaCppHandler because of missing file: _build/test/resources/examples/llamacpp/llamacpp_handler/llama-2-7b-chat.Q5_0.gguf
[  SKIPPED ] ModelPredictTest.TestLoadPredictLlamaCppHandler (0 ms)
[ RUN      ] ModelPredictTest.TestLoadPredictAotInductorResnetHandler
I0217 05:32:12.329823 138546 log_metric.cc:92] [METRICS]HandlerTime.Milliseconds:31.501974|#ModelName:resnetcppaot,Level:Model|#hostname:ip-172-31-35-108,1708147932,resnet_ts_0,resnet_ts_1
I0217 05:32:12.329858 138546 log_metric.cc:92] [METRICS]PredictionTime.Milliseconds:31.501974|#ModelName:resnetcppaot,Level:Model|#hostname:ip-172-31-35-108,1708147932,resnet_ts_0,resnet_ts_1
[       OK ] ModelPredictTest.TestLoadPredictAotInductorResnetHandler (83 ms)
[ RUN      ] ModelPredictTest.TestLoadPredictBaseHandler
I0217 05:32:12.347453 138546 log_metric.cc:92] [METRICS]HandlerTime.Milliseconds:4.545088|#ModelName:mnist_scripted_v2,Level:Model|#hostname:ip-172-31-35-108,1708147932,mnist_ts_0,mnist_ts_1
I0217 05:32:12.347479 138546 log_metric.cc:92] [METRICS]PredictionTime.Milliseconds:4.545088|#ModelName:mnist_scripted_v2,Level:Model|#hostname:ip-172-31-35-108,1708147932,mnist_ts_0,mnist_ts_1
[       OK ] ModelPredictTest.TestLoadPredictBaseHandler (14 ms)
[ RUN      ] ModelPredictTest.TestLoadPredictMnistHandler
I0217 05:32:12.361207 138546 log_metric.cc:92] [METRICS]HandlerTime.Milliseconds:3.986946|#ModelName:mnist_scripted_v2,Level:Model|#hostname:ip-172-31-35-108,1708147932,mnist_ts_0,mnist_ts_1
I0217 05:32:12.361233 138546 log_metric.cc:92] [METRICS]PredictionTime.Milliseconds:3.986946|#ModelName:mnist_scripted_v2,Level:Model|#hostname:ip-172-31-35-108,1708147932,mnist_ts_0,mnist_ts_1
[       OK ] ModelPredictTest.TestLoadPredictMnistHandler (13 ms)
[ RUN      ] ModelPredictTest.TestBackendInitWrongModelDir
E0217 05:32:12.361844 138546 model_archive.cc:53] Failed to init Manifest from: _build/test/resources/examples/mnist/MAR-INF/MANIFEST.json
[       OK ] ModelPredictTest.TestBackendInitWrongModelDir (0 ms)
[ RUN      ] ModelPredictTest.TestBackendInitWrongHandler
[       OK ] ModelPredictTest.TestBackendInitWrongHandler (0 ms)
[ RUN      ] ModelPredictTest.TestLoadModelFailure
E0217 05:32:12.367962 138546 torch_scripted_handler.cc:22] loading the model: mnist_scripted_v2, device id: -1, error: open file failed because of errno 2 on fopen: No such file or directory, file path: _build/test/resources/examples/mnist/wrong_model/mnist_script.pt
[       OK ] ModelPredictTest.TestLoadModelFailure (5 ms)
[ RUN      ] ModelPredictTest.TestLoadPredictMnistHandlerFailure
E0217 05:32:12.381099 138546 base_handler.cc:154] Failed to load tensor for request id: mnist_ts_0, c10 error: PytorchStreamReader failed reading zip archive: failed finding central directory
E0217 05:32:12.385587 138546 base_handler.cc:154] Failed to load tensor for request id: mnist_ts_1, c10 error: PytorchStreamReader failed reading zip archive: failed finding central directory
E0217 05:32:12.387691 138546 base_handler.cc:51] Failed to handle this batch after: Preprocessing
[       OK ] ModelPredictTest.TestLoadPredictMnistHandlerFailure (19 ms)
[----------] 10 tests from ModelPredictTest (2595 ms total)

[----------] 1 test from DLLoaderTest
[ RUN      ] DLLoaderTest.TestGetInstance
[       OK ] DLLoaderTest.TestGetInstance (0 ms)
[----------] 1 test from DLLoaderTest (0 ms total)

[----------] 3 tests from LoggingTest
[ RUN      ] LoggingTest.TestIncorrectLogInitialization
[       OK ] LoggingTest.TestIncorrectLogInitialization (0 ms)
[ RUN      ] LoggingTest.TestJSONConfigLogInitialization
[       OK ] LoggingTest.TestJSONConfigLogInitialization (0 ms)
[ RUN      ] LoggingTest.TestFileLogInitialization
[       OK ] LoggingTest.TestFileLogInitialization (0 ms)
[----------] 3 tests from LoggingTest (0 ms total)

[----------] 6 tests from TSLogMetricTest
[ RUN      ] TSLogMetricTest.TestCounterMetric
[       OK ] TSLogMetricTest.TestCounterMetric (1 ms)
[ RUN      ] TSLogMetricTest.TestGaugeMetric
[       OK ] TSLogMetricTest.TestGaugeMetric (1 ms)
[ RUN      ] TSLogMetricTest.TestHistogramMetric
[       OK ] TSLogMetricTest.TestHistogramMetric (1 ms)
[ RUN      ] TSLogMetricTest.TestTSLogMetricEmitWithRequestId
[       OK ] TSLogMetricTest.TestTSLogMetricEmitWithRequestId (1 ms)
[ RUN      ] TSLogMetricTest.TestTSLogMetricEmitWithoutRequestId
[       OK ] TSLogMetricTest.TestTSLogMetricEmitWithoutRequestId (0 ms)
[ RUN      ] TSLogMetricTest.TestTSLogMetricEmitWithIncorrectDimensionData
[       OK ] TSLogMetricTest.TestTSLogMetricEmitWithIncorrectDimensionData (0 ms)
[----------] 6 tests from TSLogMetricTest (7 ms total)

[----------] 2 tests from TSLogMetricsCacheTest
[ RUN      ] TSLogMetricsCacheTest.TestInitialize
[       OK ] TSLogMetricsCacheTest.TestInitialize (3 ms)
[ RUN      ] TSLogMetricsCacheTest.TestGetMetric
I0217 05:32:12.400796 138546 log_metric.cc:89] [METRICS]GaugeTsMetricExample.Count:1.5|#model_name:model_name,host_name:host_name|#hostname:ip-172-31-35-108,1708147932
[       OK ] TSLogMetricsCacheTest.TestGetMetric (1 ms)
[----------] 2 tests from TSLogMetricsCacheTest (5 ms total)

[----------] 3 tests from RegistryTest
[ RUN      ] RegistryTest.TestValidConfigFile
[       OK ] RegistryTest.TestValidConfigFile (1 ms)
[ RUN      ] RegistryTest.TestInvalidConfigFile
[       OK ] RegistryTest.TestInvalidConfigFile (0 ms)
[ RUN      ] RegistryTest.TestReInitialize
[       OK ] RegistryTest.TestReInitialize (2 ms)
[----------] 3 tests from RegistryTest (4 ms total)

[----------] 3 tests from UnitsTest
[ RUN      ] UnitsTest.TestGetExistingUnitMapping
[       OK ] UnitsTest.TestGetExistingUnitMapping (0 ms)
[ RUN      ] UnitsTest.TestGetNonExistentUnitMapping
[       OK ] UnitsTest.TestGetNonExistentUnitMapping (0 ms)
[ RUN      ] UnitsTest.TestGetEmptyUnitMapping
[       OK ] UnitsTest.TestGetEmptyUnitMapping (0 ms)
[----------] 3 tests from UnitsTest (0 ms total)

[----------] 10 tests from YAMLConfigTest
[ RUN      ] YAMLConfigTest.TestLoadValidConfigFrontendContext
[       OK ] YAMLConfigTest.TestLoadValidConfigFrontendContext (1 ms)
[ RUN      ] YAMLConfigTest.TestLoadValidConfigBackendContext
[       OK ] YAMLConfigTest.TestLoadValidConfigBackendContext (1 ms)
[ RUN      ] YAMLConfigTest.TestLoadMinimalValidConfig
[       OK ] YAMLConfigTest.TestLoadMinimalValidConfig (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadInvalidConfigWithDuplicateDimension
[       OK ] YAMLConfigTest.TestLoadInvalidConfigWithDuplicateDimension (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadInvalidConfigWithEmptyDimension
[       OK ] YAMLConfigTest.TestLoadInvalidConfigWithEmptyDimension (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadInvalidConfigWithUndefinedDimension
[       OK ] YAMLConfigTest.TestLoadInvalidConfigWithUndefinedDimension (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadInvalidConfigWithDuplicateMetricDimension
[       OK ] YAMLConfigTest.TestLoadInvalidConfigWithDuplicateMetricDimension (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadInvalidConfigWithMissingMetricName
E0217 05:32:12.410185 138546 yaml_config.cc:203] Configuration for a metric must consist of "name", "unit" and "dimensions"
[       OK ] YAMLConfigTest.TestLoadInvalidConfigWithMissingMetricName (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadInvalidConfigWithEmptyMetricName
E0217 05:32:12.410630 138546 yaml_config.cc:215] Configuration for a metric must consist of a non-empty "name"
[       OK ] YAMLConfigTest.TestLoadInvalidConfigWithEmptyMetricName (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadInvalidConfigWithDuplicateMetricName
[       OK ] YAMLConfigTest.TestLoadInvalidConfigWithDuplicateMetricName (0 ms)
[----------] 10 tests from YAMLConfigTest (6 ms total)

[----------] 1 test from ManifestTest
[ RUN      ] ManifestTest.TestInitialize
[       OK ] ManifestTest.TestInitialize (0 ms)
[----------] 1 test from ManifestTest (0 ms total)

[----------] Global test environment tear-down
[==========] 48 tests from 11 test suites ran. (2704 ms total)
[  PASSED  ] 47 tests.
[  SKIPPED ] 1 test, listed below:
[  SKIPPED ] ModelPredictTest.TestLoadPredictLlamaCppHandler

Checklist:

  • Did you have fun?
  • Have you added tests that prove your fix is effective or that this feature works?
  • Has code been commented, particularly in hard-to-understand areas?
  • Have you made corresponding changes to the documentation?

@lxning lxning changed the title [WIP] Resnet aot AOTInductor Resnet CPP example Feb 17, 2024
mreso
mreso approved these changes Feb 21, 2024
View reviewed changes
Copy link
Collaborator

@mreso mreso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, left a comment regarding include placement in header file

examples/cpp/aot_inductor/resnet/src/resnet_handler.hh
#include <folly/FileUtil.h>
#include <folly/dynamic.h>
#include <folly/json.h>
#include <fmt/format.h>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to import only header files inside the header which are actually necessary. Everything else should be imported inside the cpp file.

@mreso mreso mentioned this pull request Feb 21, 2024
Merged
8 tasks
@lxning lxning added this pull request to the merge queue Feb 22, 2024
Merged via the queue into master with commit 2f1f358 Feb 22, 2024
15 checks passed
@chauhang chauhang added this to the v0.10.0 milestone Feb 27, 2024
@chauhang chauhang added the c++ label Feb 27, 2024
@lxning lxning self-assigned this Feb 27, 2024
@lxning lxning added documentation Improvements or additions to documentation enhancement New feature or request example p0 high priority labels Feb 27, 2024
muthuraj-i2i pushed a commit to muthuraj-i2i/serve that referenced this pull request Mar 1, 2024
@lxning @mreso @andyi2it
AOTInductor Resnet CPP example (pytorch#2944)
983f886 
* fmt

* fix compile error

* fmt

* fmt

* fmt

* fix path

* remove sentencepiece

* add test file

* fmt

* fmt

* fix typo in readme

* update readme

* fix lint

* fix typo

* add model-config.yaml

---------

Co-authored-by: Matthias Reso <13337103+mreso@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mreso mreso mreso approved these changes

@chauhang chauhang Awaiting requested review from chauhang

Assignees

@lxning lxning

Labels
c++ documentation Improvements or additions to documentation enhancement New feature or request example p0 high priority
Projects
v0.10.0 lifecycle
Status: Done
Milestone
v0.10.0
Development

Successfully merging this pull request may close these issues.
3 participants
@lxning @mreso @chauhang