Remove folly dependency + fix aot on gpu #3006

mreso · 2024-03-07T07:03:08Z

Description

This PR removes folly from the cpp backend and replaces the functionality with smaller specialized libraries.
Through this move we're able to use the pip installed Pytorch and do not need to reinstall a separate libtorch on Linux.
It also fixes an issue with aot inductor examples on gpu and enables the usage of 2.3 nightlies.

Fixes #(issue)

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
This change requires a documentation update

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

Test A

torchserve_cpp build is complete. To run unit test:   ./_build/test/torchserve_cpp_test
Running main() from /home/ubuntu/serve/cpp/_build/_deps/googletest-src/googletest/src/gtest_main.cc
[==========] Running 49 tests from 12 test suites.
[----------] Global test environment set-up.
[----------] 1 test from BackendIntegTest
[ RUN      ] BackendIntegTest.TestOTFProtocolAndHandler
[       OK ] BackendIntegTest.TestOTFProtocolAndHandler (308 ms)
[----------] 1 test from BackendIntegTest (308 ms total)

[----------] 8 tests from OTFMessageTest
[ RUN      ] OTFMessageTest.TestRetieveCmd
[       OK ] OTFMessageTest.TestRetieveCmd (0 ms)
[ RUN      ] OTFMessageTest.TestEncodeLoadModelResponse
[       OK ] OTFMessageTest.TestEncodeLoadModelResponse (0 ms)
[ RUN      ] OTFMessageTest.TestUTF8EncodeLoadModelResponse
[       OK ] OTFMessageTest.TestUTF8EncodeLoadModelResponse (0 ms)
[ RUN      ] OTFMessageTest.TestRetrieveMsgLoadGpu
[       OK ] OTFMessageTest.TestRetrieveMsgLoadGpu (0 ms)
[ RUN      ] OTFMessageTest.TestRetrieveMsgLoadNoGpu
[       OK ] OTFMessageTest.TestRetrieveMsgLoadNoGpu (0 ms)
[ RUN      ] OTFMessageTest.TestEncodeSuccessInferenceResponse
[       OK ] OTFMessageTest.TestEncodeSuccessInferenceResponse (0 ms)
[ RUN      ] OTFMessageTest.TestEncodeFailureInferenceResponse
[       OK ] OTFMessageTest.TestEncodeFailureInferenceResponse (0 ms)
[ RUN      ] OTFMessageTest.TestRetrieveInferenceMsg
[       OK ] OTFMessageTest.TestRetrieveInferenceMsg (0 ms)
[----------] 8 tests from OTFMessageTest (0 ms total)

[----------] 11 tests from ModelPredictTest
[ RUN      ] ModelPredictTest.TestLoadPredictBabyLlamaHandler
[----------] 11 tests from ModelPredictTest                                                                                                                                                                                                                                           [76/1893]
[ RUN      ] ModelPredictTest.TestLoadPredictBabyLlamaHandler
Total number of tokens generated: 332
Achieved tok per sec: 229.758
Generated String:  Hello my name is
The little girl, who was three years old, was playing in the garden. She saw a big, red tomato and wanted to pick it. She reached out her hand and grabbed it.
Suddenly, a voice said, "Hey! That's my tomato!"
The little girl looked up and saw a big, angry bird. She was scared and started to cry.
The bird said, "That tomato is mine! I'm going to eat it!"
The little girl was very scared and ran away. She was so sad that she didn't get to eat the tomato.
The bird flew away and the little girl was left alone in the garden. She was very sad and cried all the way home.
<s>

Generated String:  Hello my name is
The little girl, who was three years old, was playing in the garden. She saw a big, red tomato and wanted to pick it. She reached out her hand and grabbed it.
Suddenly, a voice said, "Hey! That's my tomato!"
The little girl looked up and saw a big, angry bird. She was scared and started to cry.
The bird said, "That tomato is mine! I'm going to eat it!"
The little girl was very scared and ran away. She was so sad that she didn't get to eat the tomato.
The bird flew away and the little girl was left alone in the garden. She was very sad and cried all the way home.
<s>

[       OK ] ModelPredictTest.TestLoadPredictBabyLlamaHandler (1711 ms)
[ RUN      ] ModelPredictTest.TestLoadPredictAotInductorLlamaHandler
[       OK ] ModelPredictTest.TestLoadPredictAotInductorLlamaHandler (3176 ms)
[ RUN      ] ModelPredictTest.TestLoadPredictLlamaCppHandler
/home/ubuntu/serve/cpp/test/examples/examples_test.cc:53: Skipped
Skipping TestLoadPredictLlamaCppHandler because of missing file: _build/test/resources/examples/llamacpp/llamacpp_handler/llama-2-7b-chat.Q5_0.gguf
[  SKIPPED ] ModelPredictTest.TestLoadPredictLlamaCppHandler (0 ms)
[ RUN      ] ModelPredictTest.TestLoadPredictAotInductorBertHandler
[       OK ] ModelPredictTest.TestLoadPredictAotInductorBertHandler (2870 ms)
[ RUN      ] ModelPredictTest.TestLoadPredictAotInductorResnetHandler
[       OK ] ModelPredictTest.TestLoadPredictAotInductorResnetHandler (1428 ms)
[ RUN      ] ModelPredictTest.TestLoadPredictBaseHandler
[       OK ] ModelPredictTest.TestLoadPredictBaseHandler (37 ms)
[ RUN      ] ModelPredictTest.TestLoadPredictMnistHandler
[       OK ] ModelPredictTest.TestLoadPredictMnistHandler (17 ms)
[ RUN      ] ModelPredictTest.TestBackendInitWrongModelDir
[       OK ] ModelPredictTest.TestBackendInitWrongModelDir (1 ms)
[ RUN      ] ModelPredictTest.TestBackendInitWrongHandler
[       OK ] ModelPredictTest.TestBackendInitWrongHandler (0 ms)
[ RUN      ] ModelPredictTest.TestLoadModelFailure
[       OK ] ModelPredictTest.TestLoadModelFailure (30 ms)
[ RUN      ] ModelPredictTest.TestLoadPredictMnistHandlerFailure
[       OK ] ModelPredictTest.TestLoadPredictMnistHandlerFailure (28 ms)
[----------] 11 tests from ModelPredictTest (9303 ms total)

[----------] 1 test from DLLoaderTest
[ RUN      ] DLLoaderTest.TestGetInstance
[       OK ] DLLoaderTest.TestGetInstance (0 ms)
[----------] 1 test from DLLoaderTest (0 ms total)

[----------] 1 test from JsonTest
[ RUN      ] JsonTest.TestParsingAndGetValue
[       OK ] JsonTest.TestParsingAndGetValue (0 ms)
[----------] 1 test from JsonTest (0 ms total)

[----------] 2 tests from LoggingTest
[ RUN      ] LoggingTest.TestIncorrectLogInitialization
[       OK ] LoggingTest.TestIncorrectLogInitialization (0 ms)
[ RUN      ] LoggingTest.TestFileLogInitialization
[info] 0308 17:26:45.709 /home/ubuntu/serve/cpp/test/utils/logging_test.cc:38] TestBody Test
[       OK ] LoggingTest.TestFileLogInitialization (1 ms)
[----------] 2 tests from LoggingTest (1 ms total)

[----------] 6 tests from TSLogMetricTest
[ RUN      ] TSLogMetricTest.TestCounterMetric
[info] 0308 17:26:45.710 /home/ubuntu/serve/cpp/src/utils/metrics/log_metric.cc:89] Emit [METRICS]test_metric.Milliseconds:1|#level:model,model_name:test_model|#hostname:ip-172-31-15-139,1709918805
[info] 0308 17:26:45.711 /home/ubuntu/serve/cpp/src/utils/metrics/log_metric.cc:92] Emit [METRICS]test_metric.Milliseconds:3.5|#level:model,model_name:test_model|#hostname:ip-172-31-15-139,1709918805,test_request_id
[       OK ] TSLogMetricTest.TestCounterMetric (2 ms)
[ RUN      ] TSLogMetricTest.TestGaugeMetric
[info] 0308 17:26:45.712 /home/ubuntu/serve/cpp/src/utils/metrics/log_metric.cc:89] Emit [METRICS]test_metric.Count:1|#level:model,model_name:test_model|#hostname:ip-172-31-15-139,1709918805
[info] 0308 17:26:45.712 /home/ubuntu/serve/cpp/src/utils/metrics/log_metric.cc:89] Emit [METRICS]test_metric.Count:-2|#level:model,model_name:test_model|#hostname:ip-172-31-15-139,1709918805
[info] 0308 17:26:45.712 /home/ubuntu/serve/cpp/src/utils/metrics/log_metric.cc:92] Emit [METRICS]test_metric.Count:3.5|#level:model,model_name:test_model|#hostname:ip-172-31-15-139,1709918805,test_request_id
[info] 0308 17:26:45.712 /home/ubuntu/serve/cpp/src/utils/metrics/log_metric.cc:92] Emit [METRICS]test_metric.Count:-4|#level:model,model_name:test_model|#hostname:ip-172-31-15-139,1709918805,test_request_id
[       OK ] TSLogMetricTest.TestGaugeMetric (0 ms)
[ RUN      ] TSLogMetricTest.TestHistogramMetric
[info] 0308 17:26:45.713 /home/ubuntu/serve/cpp/src/utils/metrics/log_metric.cc:89] Emit [METRICS]test_metric.Count:1|#level:model,model_name:test_model|#hostname:ip-172-31-15-139,1709918805
[info] 0308 17:26:45.713 /home/ubuntu/serve/cpp/src/utils/metrics/log_metric.cc:89] Emit [METRICS]test_metric.Count:-2|#level:model,model_name:test_model|#hostname:ip-172-31-15-139,1709918805
[info] 0308 17:26:45.713 /home/ubuntu/serve/cpp/src/utils/metrics/log_metric.cc:92] Emit [METRICS]test_metric.Count:3.5|#level:model,model_name:test_model|#hostname:ip-172-31-15-139,1709918805,test_request_id
[info] 0308 17:26:45.713 /home/ubuntu/serve/cpp/src/utils/metrics/log_metric.cc:92] Emit [METRICS]test_metric.Count:-4|#level:model,model_name:test_model|#hostname:ip-172-31-15-139,1709918805,test_request_id
[       OK ] TSLogMetricTest.TestHistogramMetric (0 ms)
[ RUN      ] TSLogMetricTest.TestTSLogMetricEmitWithRequestId
[info] 0308 17:26:45.713 /home/ubuntu/serve/cpp/src/utils/metrics/log_metric.cc:92] Emit [METRICS]test_metric.Milliseconds:1.5|#level:model,model_name:test_model|#hostname:ip-172-31-15-139,1709918805,test_request_id
[       OK ] TSLogMetricTest.TestTSLogMetricEmitWithRequestId (0 ms)
[ RUN      ] TSLogMetricTest.TestTSLogMetricEmitWithoutRequestId
[info] 0308 17:26:45.714 /home/ubuntu/serve/cpp/src/utils/metrics/log_metric.cc:89] Emit [METRICS]test_metric.Milliseconds:1.5|#level:model,model_name:test_model|#hostname:ip-172-31-15-139,1709918805
[       OK ] TSLogMetricTest.TestTSLogMetricEmitWithoutRequestId (0 ms)
[ RUN      ] TSLogMetricTest.TestTSLogMetricEmitWithIncorrectDimensionData
[       OK ] TSLogMetricTest.TestTSLogMetricEmitWithIncorrectDimensionData (0 ms)
[----------] 6 tests from TSLogMetricTest (5 ms total)

[----------] 2 tests from TSLogMetricsCacheTest
[ RUN      ] TSLogMetricsCacheTest.TestInitialize
[       OK ] TSLogMetricsCacheTest.TestInitialize (1 ms)
[ RUN      ] TSLogMetricsCacheTest.TestGetMetric
[info] 0308 17:26:45.716 /home/ubuntu/serve/cpp/src/utils/metrics/log_metric.cc:89] Emit [METRICS]GaugeTsMetricExample.Count:1.5|#model_name:model_name,host_name:host_name|#hostname:ip-172-31-15-139,1709918805
[       OK ] TSLogMetricsCacheTest.TestGetMetric (0 ms)
[----------] 2 tests from TSLogMetricsCacheTest (1 ms total)

[----------] 3 tests from RegistryTest
[ RUN      ] RegistryTest.TestValidConfigFile
[       OK ] RegistryTest.TestValidConfigFile (0 ms)
[ RUN      ] RegistryTest.TestInvalidConfigFile
[       OK ] RegistryTest.TestInvalidConfigFile (0 ms)
[ RUN      ] RegistryTest.TestReInitialize
[       OK ] RegistryTest.TestReInitialize (0 ms)
[----------] 3 tests from RegistryTest (1 ms total)

[----------] 3 tests from UnitsTest
[ RUN      ] UnitsTest.TestGetExistingUnitMapping
[       OK ] UnitsTest.TestGetExistingUnitMapping (0 ms)
[ RUN      ] UnitsTest.TestGetNonExistentUnitMapping
[       OK ] UnitsTest.TestGetNonExistentUnitMapping (0 ms)
[ RUN      ] UnitsTest.TestGetEmptyUnitMapping
[       OK ] UnitsTest.TestGetEmptyUnitMapping (0 ms)
[----------] 3 tests from UnitsTest (0 ms total)

[----------] 10 tests from YAMLConfigTest
[ RUN      ] YAMLConfigTest.TestLoadValidConfigFrontendContext
[       OK ] YAMLConfigTest.TestLoadValidConfigFrontendContext (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadValidConfigBackendContext
[       OK ] YAMLConfigTest.TestLoadValidConfigBackendContext (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadMinimalValidConfig
[       OK ] YAMLConfigTest.TestLoadMinimalValidConfig (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadInvalidConfigWithDuplicateDimension
[       OK ] YAMLConfigTest.TestLoadInvalidConfigWithDuplicateDimension (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadInvalidConfigWithEmptyDimension
[       OK ] YAMLConfigTest.TestLoadInvalidConfigWithEmptyDimension (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadInvalidConfigWithUndefinedDimension
[       OK ] YAMLConfigTest.TestLoadInvalidConfigWithUndefinedDimension (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadInvalidConfigWithDuplicateMetricDimension
[       OK ] YAMLConfigTest.TestLoadInvalidConfigWithDuplicateMetricDimension (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadInvalidConfigWithMissingMetricName
[error] 0308 17:26:45.720 /home/ubuntu/serve/cpp/src/utils/metrics/yaml_config.cc:203] decode Configuration for a metric must consist of "name", "unit" and "dimensions"
[       OK ] YAMLConfigTest.TestLoadInvalidConfigWithMissingMetricName (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadInvalidConfigWithEmptyMetricName
[error] 0308 17:26:45.720 /home/ubuntu/serve/cpp/src/utils/metrics/yaml_config.cc:215] decode Configuration for a metric must consist of a non-empty "name"
[       OK ] YAMLConfigTest.TestLoadInvalidConfigWithEmptyMetricName (0 ms)
[ RUN      ] YAMLConfigTest.TestLoadInvalidConfigWithDuplicateMetricName
[       OK ] YAMLConfigTest.TestLoadInvalidConfigWithDuplicateMetricName (0 ms)
[----------] 10 tests from YAMLConfigTest (3 ms total)

[----------] 1 test from ManifestTest
[ RUN      ] ManifestTest.TestInitialize
[       OK ] ManifestTest.TestInitialize (0 ms)
[----------] 1 test from ManifestTest (0 ms total)

[----------] Global test environment tear-down
[==========] 49 tests from 12 test suites ran. (9626 ms total)
[  PASSED  ] 48 tests.
[  SKIPPED ] 1 test, listed below:
[  SKIPPED ] ModelPredictTest.TestLoadPredictLlamaCppHandler

Checklist:

Did you have fun?
Have you added tests that prove your fix is effective or that this feature works?
Has code been commented, particularly in hard-to-understand areas?
Have you made corresponding changes to the documentation?

… in frontend

examples/cpp/aot_inductor/bert/README.md

examples/cpp/aot_inductor/bert/src/bert_handler.cc

examples/cpp/aot_inductor/llama2/README.md

examples/cpp/aot_inductor/resnet/README.md

examples/cpp/babyllama/README.md

examples/cpp/llamacpp/README.md

mreso added 14 commits March 6, 2024 19:50

Adjust cpp backend integration test

2e8461a

Fix test_cpp_backend.py

88d1da9

Move some headers from logging.hh into cc

f54fb1a

Exchange folly logging against spdlog

04d5eb0

Remove default logging config in cpp worker + adjust config parameter…

2f3dbb6

… in frontend

Remove folly usage in handler

d5afb6c

Remove folly

dfdf429

Remove separate libtorch

8ec4b61

Add missing yaml file

55dcfeb

Set ld_library_path in cpp test

965e1c9

Support PT 2.3 and 2.2 in aot examples

b98c36a

Fix cpp_backend test

bed5530

Add env command to READMEs

6169bfd

Fix linter issue

a2d7587

mreso force-pushed the cpp/remove_folly_dependency branch from 1bdacee to a2d7587 Compare March 8, 2024 05:50

Merge branch 'master' into cpp/remove_folly_dependency

75bcacc

mreso changed the title ~~Remove folly dependency~~ Remove folly dependency + fix aot on gpu Mar 8, 2024

mreso marked this pull request as ready for review March 8, 2024 18:07

lxning reviewed Mar 8, 2024

View reviewed changes

mreso added 2 commits March 8, 2024 15:14

Merge branch 'master' into cpp/remove_folly_dependency

d54f0d7

Update logging config name in MANIFEST

0933c7d

lxning approved these changes Mar 9, 2024

View reviewed changes

Add note about gpu placement + simplify call to model->run

af7ee3d

mreso enabled auto-merge March 9, 2024 04:20

mreso added this pull request to the merge queue Mar 9, 2024

Merged via the queue into master with commit b92cf42 Mar 9, 2024
15 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove folly dependency + fix aot on gpu #3006

Remove folly dependency + fix aot on gpu #3006

mreso commented Mar 7, 2024 •

edited

Loading

Remove folly dependency + fix aot on gpu #3006

Remove folly dependency + fix aot on gpu #3006

Conversation

mreso commented Mar 7, 2024 • edited Loading

Description

Type of change

Feature/Issue validation/testing

Checklist:

mreso commented Mar 7, 2024 •

edited

Loading