Llama3.1 support removed?

[dhruvmullick]

System Info

A100

Who can help?

@byshiue

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Build from source https://github.com/triton-inference-server/tensorrtllm_backend
Install the latest transformers version supported by tensorrt-llm

Expected behavior

transformers 4.43.1 should be supported by tensorrt_llm, hence supporting Llama3.1.
This is as per https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct which says transformers version should be > 4.43

actual behavior

Llama3.1 is not supported because transformers needs to be <=4.42.4,>=4.38.2 for this tensorrt_llm

additional notes

As per #2008, transformers 4.43.1 was supported, allowing use of Llama3.1
However, in the latest Tensort-llm, this transformers version is not supported anymore #2094

Why is the support for Llama3.1 removed? Was there a bug?

[dhruvmullick]

Tagging @kaiyux

[stenreijers]

Running the same issue, bumped it to transformers-4.44.0 myself via pip using

pip install --upgrade transformers

Ive ignored the pip warnings

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
optimum 1.21.3 requires transformers[sentencepiece]<4.44.0,>=4.29.0, but you have transformers 4.44.0 which is incompatible.
tensorrt-llm 0.13.0.dev2024081300 requires transformers<=4.42.4,>=4.38.2, but you have transformers 4.44.0 which is incompatible.

I am able to create engines again.

[nv-guomingz]

Hi @dhruvmullick , thanks for reporting such issue.
Let me explain the background of this issue, the transformer 4.43.x has bugs on several models, e.g gpt-j.
So we dropped this version and keep the transformer version as <=4.42.4,>=4.38.2.

And we also mentioned that we need to upgrade transformer version to run llama3.1. Maybe it's not very clear at this moment, we'll try to imporve the quality of the doc.

In short, TRT-LLM doesn't remove the llama 3.1 supporting but it needs to upgrade transformer to 4.43+ for running.

[ttim]

@nv-guomingz do you plan to support it again without the need to bump transformers version again?

[dhruvmullick]

@nv-guomingz thank you for the note!
If it's still possible to build Llama3.1 engines, then I can ignore the warning that pops up. Closing the issue.

[nv-guomingz]

@nv-guomingz do you plan to support it again without the need to bump transformers version again?

Yes, we're testing the transformer 4.44.0 functionality in internal CI and we'll update transformer version if it pass the testing