[Usage]: Bad Request with multiple multimodal inputs when using vision LLM.

[mru4913]

Your current environment

The output of `python collect_env.py`

How would you like to use vllm

I have tried InterVL and MimiCPM by requesting with multiple multimodal inputs, but both failed to response and it comes with bad request error. I have done some research and noticed some VLMs like phi-3 already support such inputs. #5820. Is this feature still under construction? or Did I miss anything?

ONLINE INFER EXAMPLE

from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
  model="xxx",
  messages=[
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What are in these images? Is there any difference between them?",
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
          },
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
          },
        },
      ],
    }
  ],
  max_tokens=300,
)
print(response.choices[0])

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

The OpenAI-compatible server currently doesn't support multiple inputs yet. This will be implemented very soon by #8049.