fix null in tool type response

[wseaton]

The tool calling response was incorrectly returning a null for function type which breaks langchain's tools handling:

{'id': '9',
 'choices': [{'finish_reason': 'stop',
   'index': 0,
   'message': {'content': None,
    'role': 'assistant',
    'tool_calls': [{'id': 'call-202d73de-9231-4168-91c1-710cf6fbfd75',
      'type': None,
      'function': {'name': 'compute_square', 'arguments': '{"number":212}'}}]},
   'logprobs': None}],
 'created': 1723724369,
 'model': '/mnt/models/',
 'system_fingerprint': 'local',
 'object': 'chat.completion',
 'usage': {'completion_tokens': 23,
  'prompt_tokens': 74,
  'total_tokens': 97,
  'avg_tok_per_sec': 120.04951,
  'avg_prompt_tok_per_sec': 1396.2264,
  'avg_compl_tok_per_sec': 30.463577,
  'total_time_sec': 0.808,
  'total_prompt_time_sec': 0.053,
  'total_completion_time_sec': 0.755}}

The patch fixes this to return "type": "function"

[github-actions]

Code Metrics Report

  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 C Header                2           35           28            0            7
 Dockerfile              1           34           25            0            9
 Happy                   1          442          369            0           73
 JSON                   11          102          101            0            1
 Python                 45         1993         1695           62          236
 TOML                   20          606          535           11           60
-------------------------------------------------------------------------------
 Jupyter Notebooks       4            0            0            0            0
 |- Markdown             2           77           32           31           14
 |- Python               2          196          169            1           26
 (Total)                            273          201           32           40
-------------------------------------------------------------------------------
 Markdown               27         1894            0         1427          467
 |- BASH                 5          101           98            0            3
 |- JSON                 1           12           12            0            0
 |- Python               5           92           82            0           10
 |- Rust                 6          408          365           19           24
 |- TOML                 2           75           63            0           12
 (Total)                           2582          620         1446          516
-------------------------------------------------------------------------------
 Rust                  184        57551        52276         1007         4268
 |- Markdown            94          873           13          810           50
 (Total)                          58424        52289         1817         4318
===============================================================================
 Total                 296        62657        55029         2507         5121
===============================================================================
  

[EricLBuehler]

@wseaton let me know when this is ready for review!

[wseaton]

Building my fork now and should be integration testing it shortly!

[wseaton]

Yep, the above patch works, new inference output:

Request

{
  "model": "/mnt/models/",
  "messages": [
    {
      "role": "user",
      "content": "What is the square of 212?"
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "compute_square",
        "description": "Computes the square of a number.",
        "parameters": {
          "type": "object",
          "properties": {
            "number": {
              "type": "integer"
            }
          },
          "required": [
            "number"
          ]
        }
      }
    }
  ],
  "tool_choice": "auto"
}

Response

{
  "id": "1",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": null,
        "role": "assistant",
        "tool_calls": [
          {
            "id": "call-7519495d-a7aa-47bb-b577-9121be4d0900",
            "type": "function",
            "function": {
              "name": "compute_square",
              "arguments": "{\"number\":212}"
            }
          }
        ]
      },
      "logprobs": null
    }
  ],
  "created": 1723734130,
  "model": "/mnt/models/",
  "system_fingerprint": "local",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 23,
    "prompt_tokens": 74,
    "total_tokens": 97,
    "avg_tok_per_sec": 120.04951,
    "avg_prompt_tok_per_sec": 1396.2264,
    "avg_compl_tok_per_sec": 30.463577,
    "total_time_sec": 0.808,
    "total_prompt_time_sec": 0.053,
    "total_completion_time_sec": 0.755
  }
}

I think this is ready to merge

[EricLBuehler]

Looks good, thanks for the fix!