[Model] SiglipVisionModel ported from transformers

[ChristopherCho]

This PR implemented SiglipVisionModel for VLMs.

• Some of the pre-trained SiglipVisionModel cannot use vLLM's Attention layer.
  Therefore, I implemented alternative attention layers if vLLM's one is impossible.
• I tried vllm_flash_attn backend which doesn't work properly with CUDA Error.
  Thus, only the basic attention mechanism is working properly for now.
• Modified Paligemma to use implemented SiglipVisionModel.

FIX #6941
FIX #7144

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

• Comment /ready on the PR
• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

Just saw this. Thanks for the implementation, I'll leave the review to @ywang96 since he worked on PaliGemma.

[jeejeelee]

The calculation of attention can use the MEA (Memory Efficient Attention) ops from xformers. see: https://facebookresearch.github.io/xformers/components/ops.html

[ChristopherCho]

@jeejeelee

The calculation of attention can use the MEA (Memory Efficient Attention) ops from xformers. see: https://facebookresearch.github.io/xformers/components/ops.html

Thanks! I added xformers MEA and torch sdpa to give various options.

[jeejeelee]

Hi, thank you for your excellent work. I'd like to know if your implementation supports the following model:

import timm 
 model = timm.create_model(
                "vit_so400m_patch14_siglip_384.webli",
                pretrained=False,
                num_classes=0,
                dynamic_img_size=True,
                dynamic_img_pad=True,
            )

[ChristopherCho]

@jeejeelee
Hi, I believe that the pre-trained Siglip model vit_so400m_patch14_siglip_384.webli is just the same as the one on the Huggingface.
Also, afaik, the pre-trained Paligemma model uses the same pre-trained Siglip vision encoder.

When I tried loading the pre-trained Paligemma model with the following codes, I could successfully load and infer.

import requests
from PIL import Image
from vllm import LLM, SamplingParams

model_id = "google/paligemma-3b-mix-224"
prompt = "What is on the flower?"
image_file = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg?download=true"
image = Image.open(requests.get(image_file, stream=True).raw)

llm = LLM(model=model_id)
sampling_params = SamplingParams(
    temperature=0.0
)

input_dict = {
    "prompt": prompt,
    "multi_modal_data": {
        "image": image,
    }
}
outputs = llm.generate(input_dict, sampling_params)

print(outputs[0].outputs[0].text)

So, as the pre-trained Paligemma works fine, the pre-trained Siglip vision model should work.
But there are some caveats.

1. As the current version of the code is only for supporting VLMs, it does not contain any textual part of the Siglip model (e.g. SiglipTextTransformer, SiglipTextModel, etc.). Therefore, the full Siglip model cannot be loaded with this code.
2. The aforementioned Siglip model is the exact case in which you cannot use vLLM's attention. That is because the head size is 72 (the hidden size of the model is 1152 and the number of heads is 16), which is not in the supported head sizes.
   As I implemented the fallback logic for this case, it works fine but does not use the vLLM's paged attention.

[ywang96]

@jeejeelee Hi, I believe that the pre-trained Siglip model vit_so400m_patch14_siglip_384.webli is just the same as the one on the Huggingface. Also, afaik, the pre-trained Paligemma model uses the same pre-trained Siglip vision encoder.

When I tried loading the pre-trained Paligemma model with the following codes, I could successfully load and infer.

import requests
from PIL import Image
from vllm import LLM, SamplingParams

model_id = "google/paligemma-3b-mix-224"
prompt = "What is on the flower?"
image_file = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg?download=true"
image = Image.open(requests.get(image_file, stream=True).raw)

llm = LLM(model=model_id)
sampling_params = SamplingParams(
    temperature=0.0
)

input_dict = {
    "prompt": prompt,
    "multi_modal_data": {
        "image": image,
    }
}
outputs = llm.generate(input_dict, sampling_params)

print(outputs[0].outputs[0].text)

So, as the pre-trained Paligemma works fine, the pre-trained Siglip vision model should work. But there are some caveats.

1. As the current version of the code is only for supporting VLMs, it does not contain any textual part of the Siglip model (e.g. SiglipTextTransformer, SiglipTextModel, etc.). Therefore, the full Siglip model cannot be loaded with this code.
2. The aforementioned Siglip model is the exact case in which you cannot use vLLM's attention. That is because the head size is 72 (the hidden size of the model is 1152 and the number of heads is 16), which is not in the supported head sizes.
   As I implemented the fallback logic for this case, it works fine but does not use the vLLM's paged attention.

Hey @ChristopherCho! Thank you very much for this PR and I really appreciate it - will review this tonight!

[ywang96]

Hey @ChristopherCho - Thank you very much for the PR!

I took a first pass and left some comments. Mostly I'm wondering if we should really use vLLM's attention module in the ViT when it's only used once per sequence, and I suggest simply using the attention modules from transformers for now.

[ywang96]

Currently for ClipVisionModel, we don't use the vLLM internal Attention since the ViT encoder only runs once at the prefill time per sequence, thus I don't think there's much value leveraging a KV cache for this.

Have you seen a significant performance speedup using vLLM Attention compared to transformers Attention? If not, I think we'd rather just use the one from transformers for simplicity for now since this is not the major bottleneck in the whole inference pipeline.

[ChristopherCho]

Indeed, there weren't significant improvements by using vLLM Attention.
I believe it is due to the reason that you mentioned. It does not leverage the advantages of using a KV cache.
I removed the vLLM Attention part, but keep the log at bb570c3 for the future.

[ywang96]

Please revert this for now - if we're going to apply TP on the vision tower, we should do it in another separate PR with CLIPVisionModel together. Ideally, we should not apply a infrastructure change to only one model.

[ChristopherCho]

Reverted via dee55d0

[ywang96]

I really appreciate that you went out and implemented these (regardless if we're going to use them or not)!

[ywang96]

@ChristopherCho Now that we merged #7020 - I think there's some benefit of enabling TP for this model, given that SigLip is 400M parameters in 3B PaliGemma model.

However, could you take a look at the implementation of attention here instead of using the vLLM attention? The latter creates kv cache that I don't think we should be using for ViT when it's only run at the prefill stage.

[ChristopherCho]

@ywang96 I've removed vLLM Attention for now and utilized the basic attention mechanism, which is just the same as the SiglipAttention.

By the way, I've found that you mentioned the xformers MEA which is implemented here.
Would it be better to utilize this as the baseline attention mechanism? I've tested both the basic attention mechanism and the MEA and found that the MEA was a little bit slower in my environment.

[ywang96]

@ywang96 I've removed vLLM Attention for now and utilized the basic attention mechanism, which is just the same as the SiglipAttention.

By the way, I've found that you mentioned the xformers MEA which is implemented here. Would it be better to utilize this as the baseline attention mechanism? I've tested both the basic attention mechanism and the MEA and found that the MEA was a little bit slower in my environment.

Let's use default MHA implementation for this PR: I think if you use MEA then we need to necessarily TP the attention block (since it's using the vLLM TP layers). We can leave a TODO here and do the TP in a later PR!

[ChristopherCho]

@ywang96

Okay, I've implemented the code to use transformers SiglipAttention for now, but keep TP versions for TODO in this PR.
By the way, thank you for your comments!

[ywang96]

/ready

[ywang96]

Overall LGTM! I will just need to test this PR locally myself to make sure everything works fine!

[ywang96]

@ChristopherCho I've made a few more changes to this PR afterwards and verified it works with both TP=1 and TP>1. Thank you again for working on this!