[Model] Add multi-image support for minicpmv offline inference

[HwwwwwwwH]

Since it just need few changes outside minicpmv.py, I add multi-images support first for it. And I'm doing more tests for it.

You can use multi-image inputs as follow:

messages = [{
    "role":
    "user",
    "content":
    "(<image>./</image>)\n" + \
    "(<image>./</image>)\n" + \
    "What's the contents of these two images?"
}]
prompt = tokenizer.apply_chat_template(messages,
                                       tokenize=False,
                                       add_generation_prompt=True)

outputs = llm.generate({
    "prompt": prompt,
    "multi_modal_data": {
        "image": [image1, image2]
    }
}, sampling_params=sampling_params)

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

• Comment /ready on the PR
• Add ready label to the PR
• Enable auto-merge.

🚀

[HwwwwwwwH]

@DarkLight1337 Do you have time to review this? See if the modifications is fine for vLLM.

[DarkLight1337]

Some comments to clean up the code.

[DarkLight1337]

We can merge this after you have added tests that check the model's behaviour for multi-image input.

[HwwwwwwwH]

/ready

[DarkLight1337]

Can you move NestedInputs outside of the test functions, since they are duplicated?

[DarkLight1337]

There seems to be slight differences compared to the HF version but overall the results are reasonable.

[ywang96]

Going to make a change on the title to indicate this only works for offline inference. Thank you for the PR!

[cjfcsjt]

Thanks for implementing this! Does this support interleaved text-image reasoning?

[HwwwwwwwH]

Thanks for implementing this! Does this support interleaved text-image reasoning?

Of course! Just do not use chat_template and construct related prompt.

[seanzhang-zhichen]

support video inference with vllm openai server ? if that, can show some example?

[DarkLight1337]

support video inference with vllm openai server ? if that, can show some example?

Not supported yet. It will be addressed in another PR.

[DarkLight1337]

For now you can pass in a video via multi-image input. We already have an example of this in the docs about VLMs.