Support internvl2 chat template

[AllentDan]

No description provided.

[lvhan028]

InternVL2 defines 3 types of chat templates.
https://huggingface.co/OpenGVLab/InternVL2-26B/blob/main/conversation.py#L333
Among the currently released models, 2B/8B/26B adopt internlm2-chat while 4B uses phi3-chat.
We should consider them all.

[lvhan028]

Please fix the UT error

[lvhan028]

@zhulinJulia24 may update TC

[czczup]

Hi, I feel that it may not be necessary to implement these three chat templates separately. For InternVL2-2B/8B/26B, we can simply use the chat template of internlm2-chat. The main difference is the system prompt.

I used the following code to run the InternVL2 model, and it seems to work well based on my tests.

from lmdeploy import pipeline, TurbomindEngineConfig, ChatTemplateConfig
from lmdeploy.vl import load_image

model = 'OpenGVLab/InternVL2-2B'
system_prompt = '我是书生·万象，英文名是InternVL，是由上海人工智能实验室及多家合作单位联合开发的多模态基础模型。人工智能实验室致力于原始技术创新，开源开放，共享共创，推动科技进步和产业发展。'
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
chat_template_config = ChatTemplateConfig('internlm2-chat')
chat_template_config.meta_instruction = system_prompt
pipe = pipeline(model, chat_template_config=chat_template_config,
                backend_config=TurbomindEngineConfig(session_len=8192))
response = pipe(('describe this image', image))
print(response.text)

[czczup]

For InternVL2-4B, can the Phi3 chat template be used directly? I see there is an implementation here.

[lvhan028]

Hi, @czczup thanks for the suggestion.
But I am concerned that it may not be user-friendly, as users are required to be aware of setting the chat template.