The SearchByVectorID result may not accurate

[kevindiu]

Issue

After Vald receive the SearchByVectorID request, Vald LB Gateway will forward the UUID to the Vald Agent to perform the search.

For example we have 4 Vald Agents:

Agent	Memory
1	V1, V2
2	V1, V3
3	V2, V4
4	V4, V3

When user perform the searchByVectorID where the vector ID matches the vector V2, only Agent 1 and 3 can return the result, but the most accurate result maybe stored in Agent 2 or Agent 4.

To Reproduce:

Expected behavior:

All the Vald Agent should be able to perform the search.

Environment:

• Golang Version: 1.13.5
• Docker Version: 19.03.5
• Kubernetes Version: 1.16.3
• NGT Version: 1.8.4

[kpango]

I think it's not a bug, vald works correctly, I think it's kind of an improvement.

[kpango]

actually current version of gateway fetches vector from Backup Manager, all agent will return correct ANN response.
but this Issues problem will appear in future architecture. Fortunately I'm working on it so I will fix this problem with divide gateway tasks.