add support for tools for the ollama provider

[raffaelespazzoli]

several function calling enabled models are available in huggingface (https://huggingface.co/models?other=function+calling)
These models can be loaded in ollama via a modelfile. For example:

FROM ./Llama-2-7b-chat-hf-function-calling-v3.Q4_K.gguf

ollama create fLlama-2-7b -f Modelfile

can be used for: https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v3

At this point nothing should stop from using tools in an ollama served model.

[geoand]

@raffaelespazzoli thanks for the heads up!

Do you have any details about the API used for function calling?

[raffaelespazzoli]

I am not sure I understand the question. @geoand I was under the impression that there is no special API, it's the same conversational API. The difference is in how the model responds.
It could be model dependent, but I would expect all models to align to what ChatGPT does. But that is an assumption.

[geoand]

My question is if you have any documentation on what the HuggingFace API expects from the client when declaring available functions and what it replies when a function should be called.

[raffaelespazzoli]

mmm, this request has nothing to do with huggingface APIs....so I am lost.

[geoand]

Scratch that, replace Hugging Face with Ollama in my question above.

[geoand]

Asking because I see no mention of function calling in https://github.com/ollama/ollama/blob/main/docs/api.md

[raffaelespazzoli]

it should work with the openai compatibility layer of ollama. It wont' work with ollama native APIs at the moment. It's in their roadmap though.

[geoand]

If the compatibility layer works, then using the OpenAI module should just work.

Can you try it?

[raffaelespazzoli]

I did, it didn't really work as expected:

2024-02-15 09:51:20,882 INFO  [io.qua.lan.ope.OpenAiRestApi$OpenAiClientLogger] (vert.x-eventloop-thread-3) Request:
- method: POST
- url: http://localhost:11434/v1/chat/completions
- headers: [Accept: application/json], [Authorization: Bearer xxx], [Content-Type: application/json], [User-Agent: Resteasy Reactive Client], [content-length: 924]
- body: {"model":"fLlama-2-7b","messages":[{"role":"system","content":"You are a professional cook."},{"role":"user","content":"  add celery 10 grams\n  add vinegar 20 deciliters.\n  cook for 10 minutes in a pan at 300 celsius degrees\n"}],"temperature":1.0,"top_p":1.0,"presence_penalty":0.0,"frequency_penalty":0.0,"tools":[{"type":"function","function":{"name":"cook","description":"cook the food for a given duration at a given temperature in Celsius","parameters":{"type":"object","properties":{"duration":{"type":"object"},"temperature":{"type":"integer"}},"required":["duration","temperature"]}}},{"type":"function","function":{"name":"addIngredient","description":"add a given amount expressed in a given measure of a given ingredient to food","parameters":{"type":"object","properties":{"amount":{"type":"object"},"measure":{"type":"string"},"ingredient":{"type":"string"}},"required":["amount","measure","ingredient"]}}}]}

2024-02-15 09:51:25,245 INFO  [io.qua.lan.ope.OpenAiRestApi$OpenAiClientLogger] (vert.x-eventloop-thread-3) Response:
- status code: 200
- headers: [Content-Type: application/json], [Date: Thu, 15 Feb 2024 14:51:25 GMT], [Content-Length: 342]
- body: {"id":"chatcmpl-525","object":"chat.completion","created":1708008685,"model":"fLlama-2-7b","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"  stir occasionally until thickened, about 20-25 minutes"},"finish_reason":"stop"}],"usage":{"prompt_tokens":45,"completion_tokens":16,"total_tokens":61}}

[geoand]

Thanks for checking.

How can I try that myself?

[raffaelespazzoli]

here: https://github.com/raffaelespazzoli/ollama-java
Anyway I found that these function calling enabled model expects functions to be expressed in a way different from what chatGPT does. That is why the call does not work.
It would be cool to have access to express how the message to the LLM should be laid-out.
this seems to be the required format: https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v3#manual-prompt

[geoand]

Thanks, I'll have a closer look next week

[raffaelespazzoli]

actually the function calling layout for llama model is well explained here: https://docs.llama-api.com/essentials/function .
So it's different than from ChatGPT unfortunately.

[geoand]

👍

[geoand]

@langchain4j I don't think function calling is supported yet in the langchain4j-ollama, correct?

[langchain4j]

@geoand it is not supported by ollama itself yet, hopefully they will add it soon. We could add support on our side, but this would require heavy prompting, and will work only with a handful of models. I would wait till ollama adds this on their side as they are sitting closer to the models...

[geoand]

Understood, thanks!