External knowledge base succedded in "Retrieval Testing" but failed after attach it to an agent

[nicklhy]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

0.9.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

• Add a external knowledge base (RAGFlow) and test it with the "Retrieval Testing".
• Attach it to an agent and ask a question.

The test in "Retrieval Testing" works well like below.

But after attach it to an agent, it failed even with the exactly same query.

✔️ Expected Behavior

The agent should works well with an external knowledge base.

❌ Actual Behavior

The agent failed to retrieve info from the external knowledge base.

[dosubot]

@nicklhy Hello there! I'm here to assist you with any bugs, questions, or contributions you might have. Let's work together to solve this issue!

I found a similar closed issue that might be relevant to your problem: Knowledge Retrieval with external knowledge base. This issue was resolved by a pull request titled Fix/agent external knowledge retrieval on October 11, 2024 ^[1].

In your case, since the "Retrieval Testing" works but the agent fails, it could be related to how the agent is configured to interact with the external knowledge base. Here are a few steps you might consider:

1. Check Retrieval Settings: Ensure that the retrieval settings (Top K and Score Threshold) are correctly configured for the agent. These settings might differ from those used during the "Retrieval Testing" phase.

2. Validation Issues: The error messages you mentioned indicate validation issues with the input data format. Double-check the data format requirements for the agent and ensure that the data being retrieved from the external knowledge base complies with these requirements.

3. Debugging Tools: Utilize Dify's text retrieval testing feature to debug the recall effects under different retrieval methods and parameter configurations. This can help identify if the issue is with the retrieval method or parameters when used by the agent ^[2].

If these steps do not resolve the issue, reviewing the changes made in the pull request that fixed the similar issue might provide additional insights.

To continue talking to Dosu, mention @dosu.

[nicklhy]

PS, the external RAGFlow knowledge base was connected with the latest RAGFlow. RAGFlow's dify compatible api was defined here.

Since the "Retrieval Testing" succeeded after adding this external knowledge base in dify, I guess the problem might be in the dify side ?

[JohnJyong]

@nicklhy hi nicklhy, would you pls provide some log info and pls try it in basic-chat app or workflow ?

[nicklhy]

I build a simple workflow which use the same query to call an external RAGFlow dataset and an internal dataset (created directly in dify). Both datasets are created with the same document files. However, the external one always return empty results and the internal dataset works well as below.

The basic-chat app looks similar. It works well with an internal knowledge base as below.

After changing to the external RAGFlow knowledge base. There are no query results showed in chat log.

I checked the logs of docker-api-1, there is an error when query the external RAGFlow knowledge base.

2024-10-18 01:53:13,138.138 INFO [Thread-386 (_retriever)] [_client.py:1038] - HTTP Request: POST http://10.88.11.238:9380/api/v1/dify/retrieval "HTTP/1.1 200 OK"
Exception in thread Thread-386 (_retriever):
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/app/api/core/rag/retrieval/dataset_retrieval.py", line 472, in _retriever
    document = Document(
  File "/app/api/.venv/lib/python3.10/site-packages/pydantic/main.py", line 193, in __init__
    self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for Document
metadata
  Input should be a valid dictionary [type=dict_type, input_value='', input_type=str]
    For further information visit https://errors.pydantic.dev/2.8/v/dict_type

[JohnJyong]

thanks for your feedback @nicklhy , it's the gitbook document rendering error,
we are very sorry about this , the metadata field is not text,
In fact, the correct is dict.
the wrong document:

the correct document:

[nicklhy]

@JohnJyong

OK, I mentioned what we discussed here in RAGFlow's corresponding PR. Hope they can repair this problem soon.

[Yawen-1010]

We have corrected the help documentation, thanks for the correction.
https://docs.dify.ai/guides/knowledge-base/external-knowledge-api-documentation