Refactor Agent, add canceled status for CreateIndex API and Update VQueue impl

[kpango]

Signed-off-by: kpango kpango@vdaas.org

Description:

This PR was created to correct several implementation issues that were found in the process of investigating bugs in the Vald Agent.

1. Since Vald Agent's Create Index API currently only returns "Internal" when an error occurs, an API Status was added to investigate whether the error was caused by a timeout.
2. When Vald Agent's Create Index is called by multiple threads, the execution itself is executed one thread at a time, but multiple threads accumulate in the execution waiting loop, so we added an implementation to limit the waiting queue to one.
3. When many Insert Deletes are executed in multi-threading, the VQueue calculation order becomes extremely large and CreateIndex takes a lot of time, so the VQueue implementation was greatly revised.

Additional work:

1. added long test for ngt service
2. Added -short parameter to Makefile.d/test.mk to turn off the Long test by default.
3. made the timeout of tests executed in Makefile.d/test.mk variable, so that it can be overridden at runtime.
4. Added test assets to generate NegativeUniform vectors.

Related Issue:

Versions:

• Go Version: 1.19.4
• Docker Version: 20.10.8
• Kubernetes Version: 1.22.0
• NGT Version: 1.14.8

Checklist:

• I have read the CONTRIBUTING document and completed our CLA agreement.
• I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer:

[vdaas-ci]

[CHATOPS:HELP] ChatOps commands.

• 🙆‍♀️ /approve - approve
• 💌 /changelog - replace the PR body by changelog details
• 🍱 /format - format codes and add licenses
• ✅ /gen-test - generate test codes
• 🏷️ /label - add labels
• ⏪ /rebase - rebase main
• 🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

[codecov]

Codecov Report

Base: 50.23% // Head: 29.99% // Decreases project coverage by -20.24% ⚠️

Coverage data is based on head (6700ff3) compared to base (97eed29).
Patch coverage: 52.30% of modified lines in pull request are covered.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1892       +/-   ##
===========================================
- Coverage   50.23%   29.99%   -20.25%     
===========================================
  Files         231      365      +134     
  Lines       13464    33701    +20237     
===========================================
+ Hits         6764    10107     +3343     
- Misses       6499    23186    +16687     
- Partials      201      408      +207     

Impacted Files	Coverage Δ
cmd/tools/cli/loadtest/main.go	0.00% <ø> (ø)
hack/benchmark/assets/x1b/loader.go	0.00% <0.00%> (ø)
hack/benchmark/internal/assets/dataset.go	0.00% <ø> (ø)
hack/benchmark/internal/operation/insert.go	0.00% <ø> (ø)
hack/benchmark/internal/operation/operation.go	0.00% <ø> (ø)
hack/benchmark/internal/operation/option.go	0.00% <ø> (ø)
hack/benchmark/internal/operation/remove.go	0.00% <ø> (ø)
hack/benchmark/internal/operation/search.go	0.00% <ø> (ø)
hack/benchmark/internal/operation/util.go	0.00% <ø> (ø)
...ck/benchmark/internal/starter/gateway/vald/vald.go	0.00% <0.00%> (ø)
... and 283 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Pages

Latest commit:	6700ff3
Status:	✅  Deploy successful!
Preview URL:	https://7f848783.vald.pages.dev
Branch Preview URL:	https://investigate-create-index-fre.vald.pages.dev

View logs

[CLAassistant]

All committers have signed the CLA.

[vdaas-ci]

Profile Report

type	vald-agent-ngt	vald-lb-gateway	vald-discoverer	vald-manager-index
cpu
heap

other images

[hlts2]

LGTM