@dhuebner Thank you for the PR! Which models did you use for testing?

@planger
llama3.1 as WS agent and llama3.2 as orchestrator, why?

@dhuebner thanks, no particular reason, just out of curiosity and to know how to best test the PR. Thank you!

Great feature addition!

I tried to test this with LLama3.1 but I was not really successful (see below).
Also I noticed:

• You do not stream at all in the case of tools present? Is this intentional? For the user, they just see "generating" for quite a while and nothing happens
• Because of not using streaming, we do not show the function calls in the chat as for the open AI provider. I think this is a really nice feature showing the user what happens in a transparent way.

see how it is visualized

This was my test with Ollama:

@JonasHelming

You do not stream at all in the case of tools present? Is this intentional? For the user, they just see "generating" for quite a while and nothing happens

The reason is that Ollama doesn't seem to support tools with streaming. The document says:
tools: tools for the model to use if supported. Requires stream to be set to false

There is also no possibility to listen to e.g. onToolsCall(), means we will need to buffer the stream until we recognize if it is a tools_call and than present it to the user.

This was my test with Ollama:

I tried the questions mentioned in #14285 by @navr32 , that worked okay. I will test with more complex questions, although I don't know how to make it perform better with just calling the API provided...

@dhuebner OK that makes sense. However I think we should still add the tool call to the response so that is its rendered in the chat.

@JonasHelming

However I think we should still add the tool call to the response so that is its rendered in the chat.

How should it look like? I can send only one response during a request, or is there a special API to achieve this?

I think at the moment we cannot really represent tool calls generically in the non-streaming case.

In the non-streaming case, we can only return a LanguageModelTextResponse:

export interface LanguageModelTextResponse {
    text: string;
}

In the streaming-case we use LanguageModelStreamResponsePart which can contain tool calls:

export interface LanguageModelStreamResponsePart {
    content?: string | null;
    tool_calls?: ToolCall[];
}

These are then translated in the method AbstractStreamParsingChatAgent.parse() to ToolCallChatResponseContentImpl, which will show in the UI.

So I think we may need to extend our LanguageModelResponse model to be able to capture tool calls also for non-streaming responses and then process them in the AbstractTextToModelParsingChatAgent to translate them into ToolCallChatResponseContentImpl, so they show up in the UI.

@planger
Good idea!
But for now as we do not have this API, what is still missing in this PR?

Just an idea: Can't we map non-stream responses to streamed ones rather easily? We just pretend that there is a stream and once we have the answer of the LLM we send it as one blob. This way we could reuse the tools?

@sdirix
Okay, we can do that