Towards a new annotation-based model

[cescoffier]

Hello,

This topic has been discussed many times (briefly) in several issues (#237, #210, #42, #41, #10). This discussion aims to see if another annotation-based model is possible and what it would look like.

Current approach

Currently, declarative AI services are annotated with @RegisterAiService. In this annotation, we configure the memory, retriever, tools, moderation model, and chat model... Then, in the application.properties, we configure the LLM provider used by this service.

This design is simple and avoids having many annotations, but it also has a few limits:

• when you have multiple chat models, the selection requires a supplier. It is flexible but convoluted.
• tools and document retrievers are global to all methods of the AI service. This will likely be a more significant concern with the advanced RAG support from langchain4j.
• the system message must be repeated on each method

In this discussion, I would like to discuss a more sophisticated model.

The overall idea

The main ideas are:

• ease the selection of the LLM
• allows retriever and tools declaration on methods

A simpler @RegisterAiService

We first need to simplify the @RegisterAiService annotation to achieve these goals. The retriever and tools attributes will be deprecated and planned for removal.

AI qualifier or new RegisterAiService attribute

The second step is to allow the selection of the LLM to be more easily (without the supplier). There are several ways to do this, such as adding a "name" attribute or a dedicated qualifier (annotation):

@RegisterAiService(llmName="my-llm")
// or
@RegisterAiService
@LLM("my-llm")

This will enable multiple instance configurations of each LLM:

# Default
quarkus.langchain4j.openai.timeout=60s
quarkus.langchain4j.openai.my-llm.timeout=60s

Open Question: Should it be quarkus.langchain4j.<name>.<provider>.attribute=value, or quarkus.langchain4j.<provider>.<name>.attribute=value.

IMPORTANT: An AI service is bound to a single LLM in this proposal. Multi-model AI Service is out of scope. The workaround is to use multiple interfaces. One issue with multi-models is handling the "memory," as the included messages are specific to the LLM.

SystemMessage on the class

The system message (when supported) allows explaining the "role" or "scope" to the LLM. A system message requires a memory of at least 3 (system + user + response).

Currently, we need to configure the system message on each method of the AI service interface. It would be helpful to configure it once. It would also:

• create a natural boundary to the methods from the AiService, like a bounded context
• avoid sending multiple times the system message when we call several methods (and use the memory)
• guarantee that the system message is the same for all the methods

Tools and Retriever on method

Unlike the system message, tools and retrievers need a bit more flexibility. Adding documents to every call might not be helpful and can be misleading for the LLM. It can also be better to restrict the method allowed to access tools.

So, the idea would be to define two new annotations:

• DocumentRetriever ContentRetriever - this annotation can select the document retriever, but it might need to be changed with the advanced RAG idea emerging in langchain4j
• Toolbox - the list of classes containing methods annotated with @Tool (note: declaring a class that does not contain @Tool should be considered illegal).

The tools and document retriever would be added when the method is invoked instead of all the calls.

Example

@RegisterAiService
@SystemMessage("""
        You are MovieMuse, an AI answering questions about the top 100 movies from IMDB.
        Your response must be polite, use the same language as the question, and be relevant to the question.

        Introduce yourself with: "Hello, I'm MovieMuse, how can I help you?"
        """)
@Singleton 
public interface MovieMuse {

    @ContentRetriever((RetrieverExample.class) // Edited: was DocumentRetriever initially
    String chat(@MemoryId Object session, @UserMessage String question);
}

@RegisterAiService
@SystemMessage("You are a professional poet")
public interface MyAiService {

    @UserMessage("Write a poem about {topic}. The poem should be {lines} lines long. Then send this poem by email.")
    @Toolbox(EmailService.class)
    String writeAndSendAPoem(String topic, int lines);
    
    @UserMessage("Write a poem about {topic}. The poem should be {lines} lines long.")
    String writePoem(String topic, int lines);

Concerns

This proposal has a few drawbacks:

• it complexifies the code as it requires more checks, but the resulting model is more flexible.
• some parts are unclear - typically, everything related to RAG needs to work with the new advanced RAG idea
• it would require the creation of annotations specific to Quarkus

[geoand]

I think this make a lot of sense. The @SystemMessage one should be easy and the start is here.

The rest is totally doable and I do think it's the proper way to provide more control. My question really is, should we do it now or wait until this is really needed?

[langchain4j]

Good idea! My 2 cents:

avoid sending multiple times the system message when we call several methods (and use the memory)

System message is not added again if it is already in the memory (even if called from different methods). Only when System message is different (e.g. each method has own system message), it replaces the previous one in the history.

DocumentRetriever - this annotation can select the document retriever, but it might need to be changed with the advanced RAG idea emerging in langchain4j

BTW it is now called ContentRetriever in langchain4j (the name is chosen to accomodate for future multimodality). I would also probably avoid using "document" in the name and instead use "text segment" to be consistent with other code (e.g. DocumentSplitter, EmbeddingStoreIngestor, etc) and emphasize that is it not the whole document that is being retrieved, only a segment of it.

it would require the creation of annotations specific to Quarkus

I like many of the ideas (e.g Toolbox), so I would add these to langchain4j to benefit all users and to have more consistency with quarkus-langchain4j

[cescoffier]

ContentRetriever is way better!

[cescoffier]

I just updated the proposal with ContentRetriever

[langchain4j]

Annotation-less AI services offtopic:

There is one thing that I have removed from AI services that I am reconsidering adding back: "prompt derivation" or whatever is the better name for it. Same idea as spring data jpa derives SQL queries from method names. So you do

public interface Assistant {
    String tellMeAJoke();
}

String joke = assistant.tellMeAJoke(); // LLM prompt: "tell me a joke"

Or

public interface Assistant {
    String tellMeAJokeAbout(String topic);
}

String joke = assistant.tellMeAJokeAbout("python"); // LLM prompt: "tell me a joke about python"

Or

```java
public interface Assistant {
    String tellMeAJoke(String topic, String style);
}

String joke = assistant.tellMeAJoke("python", "ironic"); // LLM prompt: "tell me a joke \n topic: python \n style: ironic"

WDYT? Probably not very useful for prod cases, but might be impressive for demos 😆

This can also work together with @SystemMessage

[langchain4j]

The whole AI thing is bonkers 😆

[geoand]

I think that like you said, it only works for super short prompts, for anything real you would end up with a huge, unmaintainable name :)

[jmartisk]

It's magic, maybe even a bit counter-intuitive, especially to people around Java who generally like everything statically defined and being in full control of the code. Maybe some Python/JS folks would like it better.
It's not very flexible. If you want to change anything, then you have to refactor the APIs in your app instead of just updating some strings or templates.

[langchain4j]

Agreed, this is not for prod cases. Thanks for the feedback!

[edeandrea]

As someone who is a long-time Spring user, library maintainer, and framework committer, I never liked the magic that Spring Data uses by using method names to derive behavior. There is way too much magic and not enough understanding about how the magic actually works.

[geoand]

@langchain4j are you okay with the implied changes in the langchain4j annotations?

[langchain4j]

@geoand you mean addition of @Toolbox annotation? Yes.
I think @RegisterAiService and @ContentRetriever are relevant for Quarkus only.

[geoand]

Yeah @Toolbox.

I just saw that I already made the change to @SystemMessage :)

[geoand]

I was about to start implementing @ToolBox in LangChain4j, but then realized that it won't be tremendously useful as users would likely still need to provide a description for the function (and perhaps even configure the name) - therefore they would still need to use @Tool

[langchain4j]

Yet another option to dynamically select tools on each call:

AiServices.builder(Assistant.class)
                .tools(...tool definitions...)
                .toolSelector((userId, userMessage) -> Set.of("weather", ...)) <- this will be executed on each call to AI service.

[langchain4j]

@geoand yes, see here

[geoand]

Very interesting

[cescoffier]

As far as I remember (aka trying to decypher my handwritten notes...), the idea was to remove tools from the @RegistewrAiService but declare them per method. As we cannot reuse the @Tool annotation (declaring a tool), I went with @Toolbox.

[geoand]

Ah okay, so similar high level idea as @langchain4j