Which transfomer version could be used with VLLM 0.6.2?

[bash99]

VLLM 0.6.2 had just released few hours ago, it said no support multi image inference with Qwen2-VL.

I've try it, but it require the newest transformer and automatic install it.

When I start it use follow script (worked with vllm 0.6.1)

VLLM_WORKER_MULTIPROC_METHOD=spawn CUDA_VISIBLE_DEVICES=0,1 python -m vllm.entrypoints.openai.api_server --served-model-name Qwen2-VL-72B-Instruct-GPTQ-Int4 --model Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4 --port 7869 --dtype half --trust-remote-code --kv-cache-dtype fp8 -q gptq --disable-log-requests --gpu-memory-utilization 0.998 --max-model-len 24576 --max_n
um_seqs 16 -tp 2

it report error like

  File "/DaTa/.local/home/hai.li/miniforge3/envs/vllm/lib/python3.12/site-packages/vllm/config.py", line 1746, in _get_and_verify_max_len
    assert "factor" in rope_scaling
           ^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_section'}

if I return to old transformer with

pip install git+https://github.com/huggingface/transformers@21fac7abba2a37fae86106f87fcf9974fd1e3830

it report error like

  File "/DaTa/.local/home/hai.li/miniforge3/envs/vllm/lib/python3.12/site-packages/vllm/transformers_utils/configs/mll
ama.py", line 1, in <module>
    from transformers.models.mllama import configuration_mllama as mllama_hf_config
ModuleNotFoundError: No module named 'transformers.models.mllama'

[xiehust]

same issue +1

[mkaskov]

the same

[xiehust]

seems the issue is found:vllm-project/vllm#8829

[DarkLight1337]

We have just now fixed the issue in vllm-project/vllm#8837. Please install vLLM from source to resolve the config loading problem.

[verigle]

We have just now fixed the issue in vllm-project/vllm#8837. Please install vLLM from source to resolve the config loading problem.

vllm 依然不支持多张图片或视频的问答，请问是否有计划修复？

[DarkLight1337]

We have just now fixed the issue in vllm-project/vllm#8837. Please install vLLM from source to resolve the config loading problem.

vllm 依然不支持多张图片或视频的问答，请问是否有计划修复？

Multi-image input is currently supported in both offline and online inference, while video input is only supported for offline inference at the moment. If you need to pass videos via OpenAI API, you can instead provide multiple images for now. Please check the example in examples/openai_vision_api_client.py (especially the part labelled "Multi-image input inference")

[jbohnslav]

We have just now fixed the issue in vllm-project/vllm#8837. Please install vLLM from source to resolve the config loading problem.

Can we get a .post0 release for this? Installing from source is a lot more difficult.

[seetimee]

use
pip install https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
it will install the latest version of vllm.It worked for me just now

[DarkLight1337]

use pip install https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl it will install the latest version of vllm.It worked for me just now

I'd recommend against installing .whl files directly unless it's from a source that you trust.

[leonbadboy]

vllm 0.62 运行还是报这个错：Unrecognized keys in rope_scaling for 'rope_type'='default': {'mrope_section'}

[DarkLight1337]

vllm 0.62 运行还是报这个错：Unrecognized keys in rope_scaling for 'rope_type'='default': {'mrope_section'}

Please install vLLM from source to fix the issue.

[Carkham]

Hi all, I'm encountering the same error. It seems related to the rope_scaling["type"] settings in Qwen2VLConfig from the transformers library.

You can check the relevant code here:

if self.rope_scaling is not None and "type" in self.rope_scaling:
    if self.rope_scaling["type"] == "mrope":
        # self.rope_scaling["type"] = "default"
    self.rope_scaling["rope_type"] = self.rope_scaling["type"]
rope_config_validation(self, ignore_keys={"mrope_section"})

After commenting out that line, my program work well with transformers=4.45.2 and vllm=0.6.2.

[Tongjilibo]

change type to rope_type in config.json can fix this error in vllm==0.6.2