feature: support qwen2.5 fuction_call

[akai-shuuichi]

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

#2610 的后续，支持了Qwen2.5的fuctioncall，参考了 #2147 和Qwen2.5的chatTemplate tokenizer_config.json

Modification

    modified:   lmdeploy/model.py
    modified:   lmdeploy/serve/async_engine.py
    modified:   lmdeploy/serve/openai/api_server.py

Use cases (Optional)

基于lmdeployv0.6.2.post1

lmdeploy serve api_server  Qwen2.5-14B-Instruct-AWQ --tp 2

不携带tools参数

from openai import OpenAI
client = OpenAI(
    api_key='YOUR_API_KEY',
    base_url="http://0.0.0.0:23333/v1"
)
model_name = client.models.list().data[0].id
response = client.chat.completions.create(
  model=model_name,
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": " provide three suggestions about time management"},
  ],
    temperature=0.8,
    top_p=0.8
)
print(response)

得到类似的结果：

ChatCompletion(id='2', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content="Certainly! Effective time management can greatly enhance productivity and reduce stress. Here are three suggestions for better time management:\n\n1. **Prioritize Tasks with a To-Do List**: Start your day by listing all the tasks you need to accomplish. Then prioritize these tasks based on their importance and urgency. This helps you focus on what truly matters and prevents you from getting overwhelmed by a long list of tasks. Tools like digital planners or simple pen-and-paper lists can be very effective.\n\n2. **Implement the Pomodoro Technique**: This technique involves working for a set period (typically 25 minutes), followed by a short break (5 minutes). After four such cycles, take a longer break (15-30 minutes). This method can help maintain high levels of concentration and prevent burnout. It's particularly useful for tasks that require a lot of mental effort.\n\n3. **Use Time Blocking**: Allocate specific blocks of time for different activities throughout your day. For example, you might dedicate the early morning hours to tasks that require the most concentration, mid-morning to meetings or calls, and later in the day to less demanding tasks or administrative work. This approach helps in managing time more efficiently and can also help in maintaining a better work-life balance.\n\nBy implementing these strategies, you can improve your time management skills and achieve more in less time.", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1731635562, model='Qwen2.5-14B-Instruct-AWQ', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=271, prompt_tokens=35, total_tokens=306, completion_tokens_details=None, prompt_tokens_details=None))

实际拼接好的prompt(qwen2.5系列的sysprompt和之前的系列不一样)

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
 provide three suggestions about time management<|im_end|>
<|im_start|>assistant

携带tools参数:

from openai import OpenAI
import json

def get_current_temperature(arguments):
    # print(json.dumps(arguments))
    if json.loads(arguments).get('location') == 'San Francisco, CA, USA':
        return "26.0­°C"
    else:
        return "27.0"

def get_temperature_date(arguments):
    if json.loads(arguments)['location'] == 'San Francisco, CA, USA' and json.loads(arguments)['location'] == 'San Francisco, CA, USA':
        return "26.5­°C"
    else:
        return "27.5­°C"

tools = [{
    'type': 'function',
    'function': {
        'name': 'get_current_temperature',
        'description': 'Get current temperature at a location.',
        'parameters': {
            'type': 'object',
            'properties': {
                'location': {
                    'type': 'string',
                    'description': 'The location to get the temperature for, in the format \'City, State, Country\'.'
                },
                'unit': {
                    'type': 'string',
                    'enum': [
                        'celsius',
                        'fahrenheit'
                    ],
                    'description': 'The unit to return the temperature in. Defaults to \'celsius\'.'
                }
            },
            'required': [
                'location'
            ]
        }
    }
}, {
    'type': 'function',
    'function': {
        'name': 'get_temperature_date',
        'description': 'Get temperature at a location and date.',
        'parameters': {
            'type': 'object',
            'properties': {
                'location': {
                    'type': 'string',
                    'description': 'The location to get the temperature for, in the format \'City, State, Country\'.'
                },
                'date': {
                    'type': 'string',
                    'description': 'The date to get the temperature for, in the format \'Year-Month-Day\'.'
                },
                'unit': {
                    'type': 'string',
                    'enum': [
                        'celsius',
                        'fahrenheit'
                    ],
                    'description': 'The unit to return the temperature in. Defaults to \'celsius\'.'
                }
            },
            'required': [
                'location',
                'date'
            ]
        }
    }
}]
messages = [{'role': 'user', 'content': 'Today is 2024-11-14, What\'s the temperature in San Francisco now? How about tomorrow?'}]

client = OpenAI(api_key='YOUR_API_KEY', base_url='http://0.0.0.0:23333/v1')
model_name = client.models.list().data[0].id
response = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0.8,
    top_p=0.8,
    stream=False,
    tools=tools)
print(response.choices[0].message.tool_calls)
messages.append(response.choices[0].message)

for tool_call in response.choices[0].message.tool_calls:
    if tool_call.function.name == 'get_temperature_date':
        content = get_temperature_date(tool_call.function.arguments)
    if tool_call.function.name == 'get_current_temperature':
        content = get_current_temperature(tool_call.function.arguments)
    messages.append({
            'role': 'tool',
            'name': tool_call.function.name,
            'content': content,
            'tool_call_id': tool_call.id
        })


response = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0.8,
    top_p=0.8,
    stream=False,
    tools=tools)
print(response.choices[0].message.content)

result：

[ChatCompletionMessageToolCall(id='0', function=Function(arguments='{"location": "San Francisco, CA, USA"}', name='get_current_temperature'), type='function'),
ChatCompletionMessageToolCall(id='1', function=Function(arguments='{"location": "San Francisco, CA, USA", "date": "2024-11-15", "unit": "celsius"}', name='get_temperature_date'), type='function')]

The current temperature in San Francisco is 26.0°C. The temperature is predicted to be 26.5°C tomorrow, November 15, 2024. haha

最后一次调用的prompt：

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"description": "Get current temperature at a location.", "name": "get_current_temperature", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format 'City, State, Country'."}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The unit to return the temperature in. Defaults to 'celsius'."}}, "required": ["location"]}}
{"description": "Get temperature at a location and date.", "name": "get_temperature_date", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format 'City, State, Country'."}, "date": {"type": "string", "description": "The date to get the temperature for, in the format 'Year-Month-Day'."}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The unit to return the temperature in. Defaults to 'celsius'."}}, "required": ["location", "date"]}}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
Today is 2024-11-14, What's the temperature in San Francisco now? How about tomorrow?<|im_end|>
<|im_start|>assistant

<tool_call>
{"name": "get_current_temperature", "arguments": {"location": "San Francisco, CA, USA"}}
</tool_call>
<tool_call>
{"name": "get_temperature_date", "arguments": {"location": "San Francisco, CA, USA", "date": "2024-11-15", "unit": "celsius"}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
26.0­°C
</tool_response>
<tool_response>
26.5­°C
</tool_response>
<|im_end|>
<|im_start|>assistant

Checklist

1. Pre-commit or other linting tools are used to fix the potential lint issues.
2. The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
3. If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
4. The documentation has been modified accordingly, like docstring or example tutorials.

Checklist

1. Pre-commit or other linting tools are used to fix the potential lint issues.
2. The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
3. If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
4. The documentation has been modified accordingly, like docstring or example tutorials.

[akai-shuuichi]

tools = [
  {
    "type": "function",
    "function": {
      "name": "get_current_weather",
      "description": "Get the current weather in a given location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA",
          },
          "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
        },
        "required": ["location"],
      },
    }
  }
]
messages = [{"role": "user", "content": "What's the weather like in Boston today?"}]

client = OpenAI(api_key='YOUR_API_KEY',base_url='http://0.0.0.0:9000/v1')
model_name = client.models.list().data[0].id
response = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0.8,
    top_p=0.8,
    stream=False,
    tools=tools)
messages.append(response.choices[0].message)
messages.append({"role": "tool", "content": "{'location': 'shanghai', 'temperature': '40', 'unit': 'celsius'}", "tool_call_id": 97102})
print(messages)
response = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0.8,
    top_p=0.8,
    stream=False,
    tools=tools)
print(response)

result:

[{'role': 'user', 'content': "What's the weather like in Boston today?"}, ChatCompletionMessage(content='', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='0', function=Function(arguments='{"location": "Boston, MA", "unit": "fahrenheit"}', name='get_current_weather'), type='function')]), {'role': 'tool', 'content': "{'location': 'shanghai', 'temperature': '40', 'unit': 'celsius'}", 'tool_call_id': 97102}]
ChatCompletion(id='2', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content="It seems like the response was for Shanghai instead of Boston. Let's try getting the weather for Boston specifically.\n\n", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='0', function=Function(arguments='{"location": "Boston, MA", "unit": "fahrenheit"}', name='get_current_weather'), type='function')]))], created=1731315568, model='Qwen2.5-14B-Instruct-AWQ', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=50, prompt_tokens=245, total_tokens=295, completion_tokens_details=None, prompt_tokens_details=None))

[lvhan028]

Please resolve the linting issue

pip install pre-commit
cd lmdeploy  # the root dir of lmdeploy repo
pre-commit install
pre-commit run --all-files

[akai-shuuichi]

存在一些NPE, 解决中。

[akai-shuuichi]

目前的tools似乎只支持单个调用，我是否应该在pr修改支持同时调用多个函数？

[AllentDan]

不是每个模型都支持多函数，你可以为 qwen2.5 添加支持

[akai-shuuichi]

不是每个模型都支持多函数，你可以为 qwen2.5 添加支持

谢谢回复！准备使用正则表达式去实现它

pattern = r"<tools>(.*?)</tools>"

tools = re.findall(pattern, text)

[akai-shuuichi]

参考QwenLM/Qwen2.5#1073

1. 预期内的返回

original text：
<tool_call>tool1_info<tool_call><tool_call>tool2_info<tool_call>

处理效果：

{
    "content": "",
    "tool_calls":[
        tool1_info,
        tool2_info
    ]
}

此外，为了处理一些在标记<tool_call>之外生成的数据，我为这几种情况做了兼容处理：

1. 多余文本出现在了最前面

original text：
text1<tool_call>tool1_info<tool_call><tool_call>tool2_info<tool_call>

2. 多余文本出现在了最后面

original text：
<tool_call>tool1_info<tool_call><tool_call>tool2_info<tool_call>text1

这两种情况的处理效果都是：

{
    "content": text1,
    "tool_calls":[
        tool1_info,
        tool2_info
    ]
}

需要注意的是，模型并不能100%保证生成结果符合预期，因此除了上面三种情况以外，还可能出现其他的问题，比如

original text：
text1<tool_call>tool1_info</tool_call>text2<tool_call>tool2_info</tool_call>text3

这种情况是没办法完美解决的，我只解析出来了text1或text3（取决于是否为空）

此外，我将parse_tool_response的返回结构改成了 str, List[tuple] 以供外部解析

[akai-shuuichi]

我遗漏了一点，在多工具调用的情况下，工具的返回顺序会影响最终的正确结果，这不会受到tool_call_id的影响，如果希望在多个tool返回顺序和调用顺序不同情况下仍能保持精准，我认为可以在prompt中添加对应的tool_call_id, 但我认为这个功能应该由用户自行实现 :)

[akai-shuuichi]

我在tests/test_lmdeploy/test_model.py中添加了test_qwen2d5测试，主要测试了无工具,单工具和多工具的单轮/多轮的prompt拼接情况，预期prompt采用了apply_chat_template生成

tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-32B-Instruct-AWQ")
text = tokenizer.apply_chat_template(messages, tools=tools, add_generation_prompt=True, tokenize=False)

[AllentDan]

LGTM