[BUG] Error after upgrading to Flowise v2.0.5 with Milvus failing

[azamiftikhar1000]

Description

I recently upgraded Flowise to version 2.0.5. After the upgrade, Milvus started to fail using the same credentials that worked in the previous Flowise version 1.8.4. I am now getting the following error message:

Error: Error searching data: {"status":{"extra_info":{},"error_code":"UnexpectedError","reason":"fail to search on QueryNode 63: queryHook execution failed: service unavailable: search params in wrong format
character 'o' looking for beginning of value","code":2,"retriable"
,"detail":"failed to search: failed to search/query delegator 63 for channel in01-0fb909e951340bb-rootcoord-dml_7_447376391097125769v0: fail to search on QueryNode 63: queryHook execution failed: service unavailable: search params in wrong format
character 'o' looking for beginning of value"},"results":[]}

Sample flow that I am testing:

[janpopovic]

I'm having the same problem. Curious if this also affects other people who use Milvus and Flowise 2.x.x?

[huaian2004]

I got the same issue with flowise 2.0.7. Here is the debug info from console.

'''
10:chain:RetrievalChainWithNoHistory > 12:retriever:VectorStoreRetriever] [1.41s] Retriever run errored with error: "Error searching data: {"status":{"extra_info":{},"error_code":"UnexpectedError","reason":"fail to search on QueryNode 2: worker(2) query failed: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: [o","code":65535,"retriable":false,"detail":"failed to search: failed to search/query delegator 2 for channel by-dev-rootcoord-dml_5_452377787821047827v0: fail to search on QueryNode 2: worker(2) query failed: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: [o"},"results":[]}

Error: Error searching data: {"status":{"extra_info":{},"error_code":"UnexpectedError","reason":"fail to search on QueryNode 2: worker(2) query failed: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: [o","code":65535,"retriable":false,"detail":"failed to search: failed to search/query delegator 2 for channel by-dev-rootcoord-dml_5_452377787821047827v0: fail to search on QueryNode 2: worker(2) query failed: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: [o"},"results":[]}
at similaritySearchVectorWithScore (/usr/local/lib/node_modules/flowise/node_modules/flowise-components/dist/nodes/vectorstores/Milvus/Milvus.js:247:15)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async vectorStore.similaritySearchVectorWithScore (/usr/local/lib/node_modules/flowise/node_modules/flowise-components/dist/nodes/vectorstores/Milvus/Milvus.js:186:20)
at async Milvus.similaritySearch (/usr/local/lib/node_modules/flowise/node_modules/@langchain/community/node_modules/@langchain/core/dist/vectorstores.cjs:108:25)
at async VectorStoreRetriever.getRelevantDocuments (/usr/local/lib/node_modules/flowise/node_modules/@langchain/community/node_modules/@langchain/core/dist/retrievers/index.cjs:72:29)
at async VectorStoreRetriever._streamIterator (/usr/local/lib/node_modules/flowise/node_modules/@langchain/community/node_modules/@langchain/core/dist/runnables/base.cjs:165:9)
at async VectorStoreRetriever.transform (/usr/local/lib/node_modules/flowise/node_modules/@langchain/community/node_modules/@langchain/core/dist/runnables/base.cjs:402:9)
at async RunnableSequence._streamIterator (/usr/local/lib/node_modules/flowise/node_modules/@langchain/core/dist/runnables/base.cjs:1208:30)"
'''

[toby1991]

+1

• flowise: flowiseai/flowise:2.1.1
• milvus: milvusdb/milvus:v2.4.11-gpu

It looks like the search_params which passing to Milvus is not correct json format.

[search_params="[{\"key\":\"anns_field\",\"value\":\"langchain_vector\"},{\"key\":\"topk\",\"value\":\"4\"},{\"key\":\"metric_type\",\"value\":\"L2\"},{\"key\":\"params\",\"value\":\"[object Object]\"}]"]

It contains a \"value\":\"[object Object]\" in it, which caused the error**

parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: '[o'"] 

Milvus said in console:

[2024/09/26 08:14:48.156 +00:00] [WARN] [proxy/impl.go:3015] ["Search failed to WaitToFinish"] [traceID=151bc25f777de00a3e9538a26ad8167d] [role=proxy] [db=default] [collection=fo] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=8204] [OutputFields="[loc,langchain_text]"] [search_params="[{\"key\":\"anns_field\",\"value\":\"langchain_vector\"},{\"key\":\"topk\",\"value\":\"4\"},{\"key\":\"metric_type\",\"value\":\"L2\"},{\"key\":\"params\",\"value\":\"[object Object]\"}]"] [ConsistencyLevel=Bounded] [useDefaultConsistency=false] [nq=1] [error="failed to search: failed to search/query delegator 7 for channel by-dev-rootcoord-dml_0_452808391727775224v0: fail to search on QueryNode 7: worker(7) query failed: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: '[o'"] [errorVerbose="failed to search: failed to search/query delegator 7 for channel by-dev-rootcoord-dml_0_452808391727775224v0: fail to search on QueryNode 7: worker(7) query failed: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: '[o'\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Execute\n  | \t/workspace/source/internal/proxy/task_search.go:552\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:474\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:553\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/root/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\nWraps: (2) failed to search\nWraps: (3) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:186\n  | [...repeated from below...]\nWraps: (4) failed to search/query delegator 7 for channel by-dev-rootcoord-dml_0_452808391727775224v0\nWraps: (5) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | \t/workspace/source/internal/proxy/task_search.go:739\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:178\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/workspace/source/internal/proxy/lb_policy.go:147\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/workspace/source/internal/proxy/lb_policy.go:215\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/root/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (6) fail to search on QueryNode 7\nWraps: (7) worker(7) query failed: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: '[o'\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) merr.milvusError"]

[Banbury]

Flowise 2.1.1 crashes completely for me, when trying to upsert a vector store:

2024-10-07 16:52:16 flowise-1  | Error: 4 DEADLINE_EXCEEDED: Deadline exceeded after 15.000s,name resolution: 0.004s,metadata filters: 0.001s,Waiting for LB pick
2024-10-07 16:52:16 flowise-1  |     at callErrorFromStatus (/usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
2024-10-07 16:52:16 flowise-1  |     at Object.onReceiveStatus (/usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/client.js:193:76)
2024-10-07 16:52:16 flowise-1  |     at /usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/call-interface.js:78:35
2024-10-07 16:52:16 flowise-1  |     at Object.onReceiveStatus (/usr/local/lib/node_modules/flowise/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/utils/Grpc.js:146:25)
2024-10-07 16:52:16 flowise-1  |     at InterceptingListenerImpl.onReceiveStatus (/usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/call-interface.js:73:23)
2024-10-07 16:52:16 flowise-1  |     at Object.onReceiveStatus (/usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)
2024-10-07 16:52:16 flowise-1  |     at Object.onReceiveStatus (/usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
2024-10-07 16:52:16 flowise-1  |     at /usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/resolving-call.js:129:78
2024-10-07 16:52:16 flowise-1  |     at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
2024-10-07 16:52:16 flowise-1  | for call at
2024-10-07 16:52:16 flowise-1  |     at ServiceClientImpl.makeUnaryRequest (/usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/client.js:161:32)
2024-10-07 16:52:16 flowise-1  |     at ServiceClientImpl.<anonymous> (/usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
2024-10-07 16:52:16 flowise-1  |     at /usr/local/lib/node_modules/flowise/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/utils/Function.js:32:31
2024-10-07 16:52:16 flowise-1  |     at new Promise (<anonymous>)
2024-10-07 16:52:16 flowise-1  |     at /usr/local/lib/node_modules/flowise/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/utils/Function.js:29:16
2024-10-07 16:52:16 flowise-1  |     at Generator.next (<anonymous>)
2024-10-07 16:52:16 flowise-1  |     at fulfilled (/usr/local/lib/node_modules/flowise/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/utils/Function.js:5:58)

[Ortes]

Same here is there any workaround ?

[Ortes]

I have submitted a PR that fix the symptoms maybe not the cause of the problem but it works #3336

[kvaranovich-ascendix]

I am still getting a similar issue with Milvus on Flowise 2.1.5

[eduardconstantin]

For me this(#3623) change solved the issue