Implement tool calling

[EricLBuehler]

No description provided.

[github-actions]

Code Metrics Report

  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 C Header                2           35           28            0            7
 Dockerfile              1           34           25            0            9
 Happy                   1          442          369            0           73
 JSON                   11          102          101            0            1
 Python                 41         1586         1368           46          172
 TOML                   19          563          497           11           55
-------------------------------------------------------------------------------
 Jupyter Notebooks       2            0            0            0            0
 |- Markdown             2           77           32           31           14
 |- Python               2          196          169            1           26
 (Total)                            273          201           32           40
-------------------------------------------------------------------------------
 Markdown               24         1822            0         1372          450
 |- BASH                 5          101           98            0            3
 |- JSON                 1           12           12            0            0
 |- Python               5           92           82            0           10
 |- Rust                 6          402          359           19           24
 |- TOML                 2           75           63            0           12
 (Total)                           2504          614         1391          499
-------------------------------------------------------------------------------
 Rust                  168        54764        49717          978         4069
 |- Markdown            90          846           13          783           50
 (Total)                          55610        49730         1761         4119
===============================================================================
 Total                 270        59348        52105         2407         4836
===============================================================================
  

[EricLBuehler]

@mitsuhiko I'm having some issues with the Jinja. The Jinja chat template (found here) fails with the error invalid operation: cannot convert map to bool (in chat_template:57).

I would really appreciate it if you could check this out.

I located the section of code which I think is the problem below:

{{- bos_token }}{%- if custom_tools is defined %}{%- set tools = custom_tools %}

To reproduce this, please git clone mistral.rs, and then run:
(with an Nvidia GPU)

git switch tool_calling
cargo run --release --features cuda -- --port 1234 plain -m meta-llama/Meta-Llama-3.1-8B-Instruct -a llama

(with a Mac)

git switch tool_calling
cargo run --release --features metal -- --port 1234 plain -m meta-llama/Meta-Llama-3.1-8B-Instruct -a llama

And then, after it is done loading, in another terminal:

python3 examples/server/tool_calling_llama_31.py

[EricLBuehler]

@mitsuhiko ok, looks like tojson needs to take kwargs - in particular, indent. Would it be OK if I open a PR?

[mitsuhiko]

@mitsuhiko ok, looks like tojson needs to take kwargs - in particular, indent. Would it be OK if I open a PR?

Yes, happy to accept patches there. Also just an issue is fine that describes the exact issue.

[EricLBuehler]

@mitsuhiko thanks for merging that!

[mitsuhiko]

The changes will go out in minijinja 2.1.1

[EricLBuehler]

The changes will go out in minijinja 2.1.1

Ah ok. I'll put out a release before then, but I'll include them in the next one!