Resilience and AI Service

[cescoffier]

This issue discusses the resilience in AI Services and their impact on the memory/context.

Context:

I'm using Granite 7B instruct, with a relatively limited context size (2048 tokens). My prompt (user message) is relatively large.
I was using @Retry on the IAService method, as the model misbehaves sometimes, and retrying improves reliability (the response time is not a factor in my context).

My AI Service calls are part of an HTTP request processing, so they are part of the request scope associated with the HTTP request.

Problem:

When retrying, the context grows, including multiple times the user message, which eventually exceeds the context size.

Let's try to describe it:

HTTP request -> 
       - AI Service call - context=[user message] -> Failure
       - AI Service call (retry) - context=[user message, user message] -> Failure
       - AI Service call (retry) - context=[user message, user message, user message] -> Failure, because of context size exceeded (unrecoverable)
       -   AI Service call (retry) - context=[user message, user message, user message, user message] -> Failure, because of context size exceeded (useless, it's not recoverable)

While my issue was on a @Retry, it may happen when using @CircuitBreaker, and so on.

Some ideas:

• We could imagine a way to manipulate the context when a retry is executed so we would not append it multiple times. However, SmallRye Fault Tolerance does not have this capability yet. We could inject an ID into the call to detect that.
• We could handle retry in the chat client. The problem is that we would also need rate limiting and circuit breakers, which would duplicate a lot of complex code.

[geoand]

We could handle retry in the chat client. The problem is that we would also need rate limiting and circuit breakers, which would duplicate a lot of complex code.

I am wondering if we can behind the scenes "move" the resilience declared on the AiService to the underlying client...

[geoand]

From the Zulip discussion, another interesting idea is smallrye/smallrye-fault-tolerance#259

[maxandersen]

shouldn't the state used to do a call avoid being mutated before the call has been completed? wouldn't that avoid the "growing" ?

[geoand]

So you are essentially proposing that the chat memory only be added to when the call succeeds, right?

That could potentially work...

[geoand]

That could potentially work...

It's actually a lot trickier than I thought because there are potentially multiple API calls that go into implementing an AI service and that add (and even remove) to / from memory

[geoand]

#764 fixes this