Support LLaVA-NeXT in Vision SFT

[qgallouedec]

Closes #1785

[kashif]

do we need to pin the transformer version to the latest?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

do we need to pin the transformer version to the latest?

Good point! let me check

[kashif]

it would be good to pin the transformer to latest due to another PR as well...

[lewtun]

do we need to pin the transformer version to the latest?

Where do you want to pin the version? I don't think we want to do this in requirements.txt, but an alternative would be to specify a min version for this specific model that is checked like we do in transformers: https://github.com/huggingface/transformers/blob/273c0afc8f0289e0cb5d18dc76ccc251504298e0/src/transformers/utils/__init__.py#L260

[kashif]

@lewtun sorry i meant >= <latest> in here: https://github.com/huggingface/trl/blob/main/setup.py#L65

[lewtun]

@lewtun sorry i meant >= <latest> in here: https://github.com/huggingface/trl/blob/main/setup.py#L65

Ah no we don't want to do that because it will lead to a lot of pain for users with conflicting transformers dependencies. In my opinion we should use the checker helper (or something similar) for architectures which are version-specific

[qgallouedec]

LLaVA-NeXT has been added in v4.40.0~235 but the example is failing before huggingface/transformers@e84012737 (not released yet)

[lewtun]

LGTM once the tests pass!