salm export trtllm

[Slyne]

What does this PR do ?

Add SALM export

Collection: [NA] (nemo multimodal export)

Changelog

• Add specific line by line info of high level changes in this PR.

Usage

Run export:
Please refer to https://github.com/Slyne/NeMo/blob/salm_trtllm_llm/examples/multimodal/speech_llm/export/README.md

Run deploy:

python3 NeMo/scripts/deploy/multimodal/deploy_triton.py \
        --modality="audio" \
        --visual_checkpoint=NeMo/examples/multimodal/speech_llm/export/output/speechllm_fc_llama2_7b_perception \
        --llm_checkpoint=NeMo/examples/multimodal/speech_llm/export/speechllm_fc_llama2_7b_llm_merged.nemo \
        --llm_model_type="llama" \
        --model_type="salm" \
        --triton_model_name="salm" \
        --max_input_len=4096 \
        --max_output_len=256 \
        --max_multimodal_len=3072 \
        --triton_model_repository=/tmp/trt_model_dir/

Run query:

python3 NeMo/scripts/deploy/multimodal/query.py \
        --model_name="salm" \
        --model_type="salm" \
        --input_text="Q: what's the transcription of the audio? A:" \
        --input_media=/ws/data/test-clean-processed/7729-102255-0009.wav

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

[Slyne]

@meatybobby

[meatybobby]

LGTM