Triton deployment improvements for in-framework models #9600

jukim-nv · 2024-07-03T07:21:11Z

What does this PR do ?

Adding better support for triton deployment of in-framework models using MegatronLLMDeployable.

Collection: [Note which collection this PR will affect]
scripts/deploy
tests/deploy

Changelog

Add NemoQueryLLMPyTorch class for easy query of MegatronLLMDeployable
Changes to nemo_export.py to accurately test MegatronLLMDeployable models
A separate deploy_inframework_triton.py deployment script to easily deploy MegatronLLMDeployable on a Triton server

Usage

python scripts/deploy/nlp/deploy_inframework_triton.py -nc <path to .nemo checkpoint> -tmn TritonModelName 

python tests/export/nemo_export.py --checkpoint_dir <path to .nemo checkpoint> --model_type gpt --min_tps 1 --max_output_len 24 --test_deployment False --in_framework True --model_name TritonModelName

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

…use in export tests

…mer_engine

…nce MegatronLLMDeployable does not support streaming add query_inframework.py since original query.py does not work with in-framework deployments

oyilmaz-nvidia

Ran this python tests/export/nemo_export.py --model_name LLAMA2-7B-base --existing_test_models --min_tps 1 --in_framework True and got the following error

[rank0]: Traceback (most recent call last):
[rank0]: File "/usr/local/lib/python3.10/dist-packages/pytriton/client/client.py", line 661, in _infer
[rank0]: response = self._infer_client.infer(
[rank0]: File "/usr/local/lib/python3.10/dist-packages/tritonclient/http/_client.py", line 1482, in infer
[rank0]: _raise_if_error(response)
[rank0]: File "/usr/local/lib/python3.10/dist-packages/tritonclient/http/_utils.py", line 69, in _raise_if_error
[rank0]: raise error
[rank0]: tritonclient.utils.InferenceServerException: [400] [request id: 0] unexpected shape for input 'max_length' for model 'LLAMA2-7B-base'. Expected [-1,1], got [1]. NOTE: Setting a non-zero max_batch_size in the model config requires a batch dimension to be prepended to each input shape. If you want to specify the full shape including the batch dim in your input dims config, try setting max_batch_size to zero. See the model configuration docs for more info on max_batch_size.

[rank0]: The above exception was the direct cause of the following exception:

[rank0]: Traceback (most recent call last):
[rank0]: File "/opt/NeMo/tests/export/nemo_export.py", line 909, in
[rank0]: run_inference_tests(args)
[rank0]: File "/opt/NeMo/tests/export/nemo_export.py", line 777, in run_inference_tests
[rank0]: result_dic[tps] = run_existing_checkpoints(
[rank0]: File "/opt/NeMo/tests/export/nemo_export.py", line 482, in run_existing_checkpoints
[rank0]: return run_in_framework_inference(
[rank0]: File "/opt/NeMo/tests/export/nemo_export.py", line 561, in run_in_framework_inference
[rank0]: output_deployed = nq.query_llm(
[rank0]: File "/opt/NeMo/nemo/deploy/nlp/query_llm.py", line 123, in query_llm
[rank0]: result_dict = client.infer_batch(**inputs)
[rank0]: File "/usr/local/lib/python3.10/dist-packages/pytriton/client/client.py", line 603, in infer_batch
[rank0]: return self._infer(inputs or named_inputs, parameters, headers)
[rank0]: File "/usr/local/lib/python3.10/dist-packages/pytriton/client/client.py", line 678, in _infer
[rank0]: raise PyTritonClientInferenceServerError(
[rank0]: pytriton.client.exceptions.PyTritonClientInferenceServerError: Error occurred during inference request. Message: [request id: 0] unexpected shape for input 'max_length' for model 'LLAMA2-7B-base'. Expected [-1,1], got [1]. NOTE: Setting a non-zero max_batch_size in the model config requires a batch dimension to be prepended to each input shape. If you want to specify the full shape including the batch dim in your input dims config, try setting max_batch_size to zero. See the model configuration docs for more info on max_batch_size.
Signal (2) received.

But this worked

python tests/export/nemo_export.py --model_name llama --model_type llama --min_tps 1 --checkpoint_dir /opt/checkpoints/LLAMA2-7B-base/LLAMA2-7B-base-1.nemo --in_framework True

Signed-off-by: jukim-nv <jukim-nv@users.noreply.github.com>

tests/export/nemo_export.py

scripts/deploy/nlp/deploy_inframework_triton.py

scripts/deploy/nlp/query_inframework.py

tests/export/nemo_export.py

jukim-nv · 2024-07-03T07:22:28Z

tests/export/nemo_export.py

            all_expected_outputs.append(expected_output)
-            all_actual_outputs.append(model_output)
+            if model is not None:
+                if isinstance(model, MegatronLLMDeployable):


I don't feel particularly good about using isinstance() here. Would be open to a better way to split the different paths.

tests/export/nemo_export.py

…amework mode

oyilmaz-nvidia · 2024-07-07T16:33:57Z

Could you please address all of these CodeQL reviews?

* add NemoQueryLLMPyTorch class for triton query of in-framework models * nemo_export.py changes to better support in-framework models * separate out in-framework version of triton deploy script * add generate() function to MegatronLLMDeployable to allow for direct use in export tests * use NemoQueryLLMPyTorch in deploy tests * add warning message for when MegatronLLMDeployable overrides transformer_engine * remove enable_streaming argument from deploy_inframework_triton.py since MegatronLLMDeployable does not support streaming add query_inframework.py since original query.py does not work with in-framework deployments * Apply isort and black reformatting Signed-off-by: jukim-nv <jukim-nv@users.noreply.github.com> * skip trtllm support check if in_framework testing * remove unused imports * run_existing_checkpoints was passing wrong prompts argument for in-framework mode * fix unused import in query_inframework.py --------- Signed-off-by: jukim-nv <jukim-nv@users.noreply.github.com> Co-authored-by: jukim-nv <jukim-nv@users.noreply.github.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com>

* add NemoQueryLLMPyTorch class for triton query of in-framework models * nemo_export.py changes to better support in-framework models * separate out in-framework version of triton deploy script * add generate() function to MegatronLLMDeployable to allow for direct use in export tests * use NemoQueryLLMPyTorch in deploy tests * add warning message for when MegatronLLMDeployable overrides transformer_engine * remove enable_streaming argument from deploy_inframework_triton.py since MegatronLLMDeployable does not support streaming add query_inframework.py since original query.py does not work with in-framework deployments * Apply isort and black reformatting Signed-off-by: jukim-nv <jukim-nv@users.noreply.github.com> * skip trtllm support check if in_framework testing * remove unused imports * run_existing_checkpoints was passing wrong prompts argument for in-framework mode * fix unused import in query_inframework.py --------- Signed-off-by: jukim-nv <jukim-nv@users.noreply.github.com> Co-authored-by: jukim-nv <jukim-nv@users.noreply.github.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com>

* add NemoQueryLLMPyTorch class for triton query of in-framework models * nemo_export.py changes to better support in-framework models * separate out in-framework version of triton deploy script * add generate() function to MegatronLLMDeployable to allow for direct use in export tests * use NemoQueryLLMPyTorch in deploy tests * add warning message for when MegatronLLMDeployable overrides transformer_engine * remove enable_streaming argument from deploy_inframework_triton.py since MegatronLLMDeployable does not support streaming add query_inframework.py since original query.py does not work with in-framework deployments * Apply isort and black reformatting Signed-off-by: jukim-nv <jukim-nv@users.noreply.github.com> * skip trtllm support check if in_framework testing * remove unused imports * run_existing_checkpoints was passing wrong prompts argument for in-framework mode * fix unused import in query_inframework.py --------- Signed-off-by: jukim-nv <jukim-nv@users.noreply.github.com> Co-authored-by: jukim-nv <jukim-nv@users.noreply.github.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: tonyjie <jl4257@cornell.edu>

* add NemoQueryLLMPyTorch class for triton query of in-framework models * nemo_export.py changes to better support in-framework models * separate out in-framework version of triton deploy script * add generate() function to MegatronLLMDeployable to allow for direct use in export tests * use NemoQueryLLMPyTorch in deploy tests * add warning message for when MegatronLLMDeployable overrides transformer_engine * remove enable_streaming argument from deploy_inframework_triton.py since MegatronLLMDeployable does not support streaming add query_inframework.py since original query.py does not work with in-framework deployments * Apply isort and black reformatting Signed-off-by: jukim-nv <jukim-nv@users.noreply.github.com> * skip trtllm support check if in_framework testing * remove unused imports * run_existing_checkpoints was passing wrong prompts argument for in-framework mode * fix unused import in query_inframework.py --------- Signed-off-by: jukim-nv <jukim-nv@users.noreply.github.com> Co-authored-by: jukim-nv <jukim-nv@users.noreply.github.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>

jukim-nv added 7 commits July 3, 2024 00:12

add NemoQueryLLMPyTorch class for triton query of in-framework models

696eb6b

nemo_export.py changes to better support in-framework models

1d9594b

separate out in-framework version of triton deploy script

a0f301f

add generate() function to MegatronLLMDeployable to allow for direct …

0e1b1cb

…use in export tests

use NemoQueryLLMPyTorch in deploy tests

4b3400d

add warning message for when MegatronLLMDeployable overrides transfor…

bdbeb94

…mer_engine

remove enable_streaming argument from deploy_inframework_triton.py si…

cbc6265

…nce MegatronLLMDeployable does not support streaming add query_inframework.py since original query.py does not work with in-framework deployments

jukim-nv force-pushed the in_framework_export branch from 7a38048 to cbc6265 Compare July 4, 2024 02:18

jukim-nv changed the title ~~Draft: Triton deployment improvements for in-framework models~~ Triton deployment improvements for in-framework models Jul 4, 2024

oyilmaz-nvidia added NLP Run CICD labels Jul 5, 2024

oyilmaz-nvidia reviewed Jul 5, 2024

View reviewed changes

merge in latest main

405aa00

github-actions bot removed the NLP label Jul 6, 2024

Apply isort and black reformatting

9af9374

Signed-off-by: jukim-nv <jukim-nv@users.noreply.github.com>

github-advanced-security bot found potential problems Jul 6, 2024

View reviewed changes

jukim-nv added 2 commits July 5, 2024 18:02

skip trtllm support check if in_framework testing

38932df

remove unused imports

e58d0e3

jukim-nv commented Jul 6, 2024

View reviewed changes

jukim-nv and others added 2 commits July 5, 2024 18:29

run_existing_checkpoints was passing wrong prompts argument for in-fr…

15ca530

…amework mode

Merge branch 'main' into in_framework_export

c103674

jukim-nv and others added 2 commits July 8, 2024 16:45

fix unused import in query_inframework.py

c6b5366

Merge branch 'main' into in_framework_export

8fdb440

oyilmaz-nvidia added NLP and removed Run CICD labels Jul 9, 2024

oyilmaz-nvidia approved these changes Jul 9, 2024

View reviewed changes

oyilmaz-nvidia added the Run CICD label Jul 9, 2024

oyilmaz-nvidia merged commit 8898b76 into NVIDIA:main Jul 9, 2024
113 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Triton deployment improvements for in-framework models #9600

Triton deployment improvements for in-framework models #9600

jukim-nv commented Jul 3, 2024 •

edited

Loading

oyilmaz-nvidia left a comment •

edited

Loading

jukim-nv Jul 3, 2024

oyilmaz-nvidia commented Jul 7, 2024

Triton deployment improvements for in-framework models #9600

Triton deployment improvements for in-framework models #9600

Conversation

jukim-nv commented Jul 3, 2024 • edited Loading

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

oyilmaz-nvidia left a comment • edited Loading

Choose a reason for hiding this comment

jukim-nv Jul 3, 2024

Choose a reason for hiding this comment

oyilmaz-nvidia commented Jul 7, 2024

jukim-nv commented Jul 3, 2024 •

edited

Loading

oyilmaz-nvidia left a comment •

edited

Loading