[Feature]: Output state configuration of vision encoder In VLM

[litianjian]

Anything you want to discuss about vllm.

When siglip or clip acts as a multimodal vision encoder, there will have several cases:

• The output state of an intermediate layer is used without layer normalization
• The output state of the last layer is used without layer normalization
• The output state of the last layer is used with layer normalization

For example, In the LLaVA-Next code implementation, post_layernorm is not used.

#8106 #8155

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.