[Feature Request]: agent RAG

[shanmu-raoyunfei]

Is your feature request related to a problem? Please describe.

The agent needs to acquire an external knowledge base to answer the user's questions

Describe the solution you'd like

Support RAG

Additional context

No response

[ekzhu]

We have a contrib agent that has RAG capability: https://microsoft.github.io/autogen/docs/reference/agentchat/contrib/retrieve_assistant_agent

Welcome new contributions!

[shanmu-raoyunfei]

It would be nice to be able to select different types of agents in the agent, such as RetrieveAssistantAgent, CompressibleAgent, MathUserProxyAgent, etc

[AlbertRapp]

@ekzhu I'm a bit confused as to what's the point of the retrieveAssistantAgent? Can you clarify the purpose of that agent? To me, It seems like only a RetrieveUserProxyAgent can actually retrieve something, i.e. a retrieveAssistantAgent cannot deal with a retrieve_config argument.

It would be great to have a retrieveAssistantAgent able to retrieve context. I'm thinking of having a group chat of multiple assistants that can retrieve different documents without having to rely on functions that can call a RetrieveUserProxyAgent (because apparently there can only be one user_proxy in a group chat).

Or am I missing something? I've been looking through the docs and examples quite a bit but I couldn't find anything that would allow me to do that.

[sonichi]

@thinkall would you like to share your thoughts?

[thinkall]

Hi @shanmu-raoyunfei , @AlbertRapp , thanks for the feedbacks.

In the current design, RetrieveUserProxyAgent is responsible for retrieving data, instead of RetrieveAssistantAgent. This is because the database, docs should be accessed only by a "User" or their proxy ("UserProxy"). "AssistantAgent" only assists in communicating with LLMs. The reason for having a "RetrieveAssistantAgent" instead of simply using the default "AssistantAgent" is that the latter cannot handle certain post-processing tasks.

We support multiple user proxy agents in group chat, here you can find examples: blog, notebook.

We are also refactoring the RAG agents to improve the usability. Please stay tuned.

[sonichi]

@thinkall could you create a roadmap issue for the RAG plans?

[hughlv]

@thinkall Is there a working example on how to use an existing external knowledge base? All the examples and docs are about vectorizing two exisitng docs and then querying on them. Can I config the retrieveuseragent to connect to a read-only pgvector db?

Vectorizing is usually not as simple as passing in a few documents, we need to review the result of the vectorization and do adjustment as needed, such as edit or delete certain chunks. This can be done in a seperate system, does not need to be the scope of AutoGen.

[thinkall]

@thinkall Is there a working example on how to use an existing external knowledge base? All the examples and docs are about vectorizing two exisitng docs and then querying on them. Can I config the retrieveuseragent to connect to a read-only pgvector db?

Vectorizing is usually not as simple as passing in a few documents, we need to review the result of the vectorization and do adjustment as needed, such as edit or delete certain chunks. This can be done in a seperate system, does not need to be the scope of AutoGen.

Hi @hughlv , sure, you can! If you pass in a pgvector client w/ the docs you needed, then it will be used.