-
Notifications
You must be signed in to change notification settings - Fork 77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add efficient search logic for large top-k users #2491
Add efficient search logic for large top-k users #2491
Conversation
Warning Rate Limit Exceeded@kpango has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 1 minutes and 1 seconds before requesting another review. How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. WalkthroughThe repository has undergone extensive updates, primarily focused on enhancing security and reliability in file downloads across various Makefiles, with the addition of Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
[CHATOPS:HELP] ChatOps commands.
|
[WARNING:INTCFG] Changes in |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2491 +/- ##
==========================================
- Coverage 18.94% 18.91% -0.03%
==========================================
Files 563 565 +2
Lines 64707 65223 +516
==========================================
+ Hits 12256 12338 +82
- Misses 51656 52087 +431
- Partials 795 798 +3 ☔ View full report in Codecov by Sentry. |
884e28b
to
2fc3e5e
Compare
Deploying vald with Cloudflare Pages
|
2fc3e5e
to
391171e
Compare
Signed-off-by: kpango <kpango@vdaas.org>
7e7960b
to
91475b4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
Out of diff range and nitpick comments (14)
docs/tutorial/vald-agent-standalone-on-k8s.md (6)
Line range hint
12-12
: Add alternate text to the image for accessibility.- <img src="../../assets/docs/tutorial/vald-agent-standalone-on-k8s.svg"> + <img src="../../assets/docs/tutorial/vald-agent-standalone-on-k8s.svg" alt="Architecture of Vald Agent Standalone on Kubernetes">
Line range hint
20-20
: Correct the link fragment to match the actual section ID.- 1. [Check and Satisfy the Requirements](#Requirements) + 1. [Check and Satisfy the Requirements](#requirements)
Line range hint
21-21
: Correct the link fragment to match the actual section ID.- 1. [Prepare Kubernetes Cluster](#Prepare-the-Kubernetes-Cluster) + 1. [Prepare Kubernetes Cluster](#prepare-the-kubernetes-cluster)
Line range hint
22-22
: Correct the link fragment to match the actual section ID.- 1. [Deploy Vald Agent Standalone on Kubernetes cluster](#Deploy-Vald-Agent-Standalone-on-Kubernetes-Cluster) + 1. [Deploy Vald Agent Standalone on Kubernetes cluster](#deploy-vald-agent-standalone-on-kubernetes-cluster)
Line range hint
23-23
: Correct the link fragment to match the actual section ID.- 1. [Run Example Code](#Run-Example-Code) + 1. [Run Example Code](#run-example-code)
Line range hint
24-24
: Correct the link fragment to match the actual section ID.- 1. [Cleanup](#Cleanup) + 1. [Cleanup](#cleanup)docs/tutorial/vald-multicluster-on-k8s.md (1)
Line range hint
15-15
: Add alternate text to the image for accessibility.- <img src="../../assets/docs/tutorial/vald-multicluster-on-k8s.png"> + <img src="../../assets/docs/tutorial/vald-multicluster-on-k8s.png" alt="Architecture of Multiple Vald Clusters using Vald Mirror Gateway">docs/tutorial/get-started.md (6)
Line range hint
15-15
: Add alternate text to the image for accessibility.- <img src="../../assets/docs/tutorial/getstarted.svg" /> + <img src="../../assets/docs/tutorial/getstarted.svg" alt="Architecture of Vald Deployment for Getting Started" />
Line range hint
19-19
: Correct the link fragment to match the actual section ID.- 1. [Check and Satisfy the Requirements](#Requirements) + 1. [Check and Satisfy the Requirements](#requirements)
Line range hint
20-20
: Correct the link fragment to match the actual section ID.- 1. [Prepare Kubernetes Cluster](#Prepare-the-Kubernetes-Cluster) + 1. [Prepare Kubernetes Cluster](#prepare-the-kubernetes-cluster)
Line range hint
21-21
: Correct the link fragment to match the actual section ID.- 1. [Deploy Vald on Kubernetes Cluster](#Deploy-Vald-on-Kubernetes-Cluster) + 1. [Deploy Vald on Kubernetes Cluster](#deploy-vald-on-kubernetes-cluster)
Line range hint
22-22
: Correct the link fragment to match the actual section ID.- 1. [Run Example Code](#Run-Example-Code) + 1. [Run Example Code](#run-example-code)
Line range hint
23-23
: Correct the link fragment to match the actual section ID.- 1. [Cleanup](#Cleanup) + 1. [Cleanup](#cleanup)pkg/gateway/lb/handler/grpc/handler.go (1)
Line range hint
468-494
: Ensure comprehensive unit tests fordoSearch
.This function is central to the search functionality and interacts with multiple components. It's crucial to have extensive unit tests covering various scenarios and configurations. Would you like assistance in setting up these tests?
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files ignored due to path filters (25)
apis/grpc/v1/agent/core/agent.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/agent/sidecar/sidecar.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/discoverer/discoverer.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/filter/egress/egress_filter.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/filter/ingress/ingress_filter.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/manager/index/index_manager.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/mirror/mirror.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/payload/payload.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/payload/payload.pb.json.go
is excluded by!**/*.pb.json.go
apis/grpc/v1/payload/payload_vtproto.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/rpc/errdetails/error_details.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/rpc/errdetails/error_details.pb.json.go
is excluded by!**/*.pb.json.go
apis/grpc/v1/vald/filter.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/vald/flush.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/vald/insert.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/vald/object.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/vald/remove.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/vald/search.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/vald/update.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
apis/grpc/v1/vald/upsert.pb.go
is excluded by!**/*.pb.go
,!**/*.pb.go
example/client/go.mod
is excluded by!**/*.mod
example/client/go.sum
is excluded by!**/*.sum
go.mod
is excluded by!**/*.mod
go.sum
is excluded by!**/*.sum
rust/Cargo.lock
is excluded by!**/*.lock
Files selected for processing (57)
- .gitfiles (8 hunks)
- .github/ISSUE_TEMPLATE/bug_report.md (1 hunks)
- .github/ISSUE_TEMPLATE/security_issue_report.md (1 hunks)
- .github/PULL_REQUEST_TEMPLATE.md (1 hunks)
- Makefile (1 hunks)
- Makefile.d/client.mk (1 hunks)
- Makefile.d/dependencies.mk (2 hunks)
- Makefile.d/functions.mk (1 hunks)
- Makefile.d/helm.mk (3 hunks)
- Makefile.d/k3d.mk (1 hunks)
- Makefile.d/k8s.mk (1 hunks)
- Makefile.d/kind.mk (1 hunks)
- Makefile.d/minikube.mk (1 hunks)
- Makefile.d/proto.mk (1 hunks)
- Makefile.d/tools.mk (3 hunks)
- apis/docs/v1/docs.md (1 hunks)
- apis/proto/v1/payload/payload.proto (2 hunks)
- apis/swagger/v1/vald/filter.swagger.json (1 hunks)
- apis/swagger/v1/vald/search.swagger.json (1 hunks)
- buf.gen.yaml (1 hunks)
- charts/vald-benchmark-operator/values.schema.json (1 hunks)
- charts/vald-helm-operator/crds/valdrelease.yaml (52 hunks)
- charts/vald/values.schema.json (195 hunks)
- charts/vald/values.yaml (6 hunks)
- dockers/tools/benchmark/job/Dockerfile (2 hunks)
- docs/tutorial/get-started.md (2 hunks)
- docs/tutorial/vald-agent-standalone-on-k8s.md (1 hunks)
- docs/tutorial/vald-multicluster-on-k8s.md (1 hunks)
- docs/user-guides/deployment.md (1 hunks)
- example/client/go.mod.default (1 hunks)
- hack/go.mod.default (2 hunks)
- internal/config/grpc.go (1 hunks)
- internal/config/grpc_test.go (1 hunks)
- internal/net/grpc/client.go (2 hunks)
- internal/net/grpc/option.go (6 hunks)
- internal/net/http/metrics/pprof.go (1 hunks)
- internal/net/http/metrics/pprof_test.go (1 hunks)
- internal/safety/safety.go (2 hunks)
- k8s/agent/ngt/configmap.yaml (3 hunks)
- k8s/discoverer/configmap.yaml (3 hunks)
- k8s/discoverer/deployment.yaml (1 hunks)
- k8s/gateway/gateway/lb/configmap.yaml (3 hunks)
- k8s/gateway/gateway/lb/deployment.yaml (1 hunks)
- k8s/manager/index/configmap.yaml (3 hunks)
- k8s/manager/index/deployment.yaml (1 hunks)
- k8s/operator/helm/crds/valdrelease.yaml (52 hunks)
- pkg/gateway/lb/handler/grpc/aggregation.go (22 hunks)
- pkg/gateway/lb/handler/grpc/handler.go (15 hunks)
- pkg/gateway/lb/handler/grpc/search_benchmark_test.go (2 hunks)
- rust/libs/proto/src/payload.v1.rs (1 hunks)
- versions/GOLANGCILINT_VERSION (1 hunks)
- versions/GO_VERSION (1 hunks)
- versions/HELM_VERSION (1 hunks)
- versions/JAEGER_OPERATOR_VERSION (1 hunks)
- versions/K3S_VERSION (1 hunks)
- versions/KUBECTL_VERSION (1 hunks)
- versions/PROMETHEUS_STACK_VERSION (1 hunks)
Files not processed due to max files limit (16)
- versions/REVIEWDOG_VERSION
- versions/actions/ACTIONS_CHECKOUT
- versions/actions/ACTIONS_DOWNLOAD_ARTIFACT
- versions/actions/ACTIONS_SETUP_GO
- versions/actions/ACTIONS_UPLOAD_ARTIFACT
- versions/actions/AZURE_SETUP_HELM
- versions/actions/CODECOV_CODECOV_ACTION
- versions/actions/DOCKER_SETUP_BUILDX_ACTION
- versions/actions/GITHUB_CODEQL_ACTION_ANALYZE
- versions/actions/GITHUB_CODEQL_ACTION_AUTOBUILD
- versions/actions/GITHUB_CODEQL_ACTION_INIT
- versions/actions/GITHUB_CODEQL_ACTION_UPLOAD_SARIF
- versions/actions/GITHUB_ISSUE_METRICS
- versions/actions/REVIEWDOG_ACTION_HADOLINT
- versions/actions/REVIEWDOG_ACTION_LANGUAGETOOL
- versions/actions/SHOGO82148_ACTIONS_UPLOAD_RELEASE_ASSET
Files not summarized due to errors (1)
- charts/vald/values.schema.json: Error: Message exceeds token limit
Files not reviewed due to errors (1)
- pkg/gateway/lb/handler/grpc/aggregation.go (no review received)
Files skipped from review due to trivial changes (1)
- dockers/tools/benchmark/job/Dockerfile
Files skipped from review as they are similar to previous changes (49)
- .gitfiles
- .github/ISSUE_TEMPLATE/bug_report.md
- .github/ISSUE_TEMPLATE/security_issue_report.md
- .github/PULL_REQUEST_TEMPLATE.md
- Makefile
- Makefile.d/client.mk
- Makefile.d/dependencies.mk
- Makefile.d/functions.mk
- Makefile.d/helm.mk
- Makefile.d/k3d.mk
- Makefile.d/k8s.mk
- Makefile.d/kind.mk
- Makefile.d/minikube.mk
- Makefile.d/proto.mk
- Makefile.d/tools.mk
- apis/proto/v1/payload/payload.proto
- apis/swagger/v1/vald/filter.swagger.json
- apis/swagger/v1/vald/search.swagger.json
- buf.gen.yaml
- charts/vald-benchmark-operator/values.schema.json
- charts/vald-helm-operator/crds/valdrelease.yaml
- charts/vald/values.yaml
- docs/user-guides/deployment.md
- example/client/go.mod.default
- hack/go.mod.default
- internal/config/grpc.go
- internal/config/grpc_test.go
- internal/net/grpc/client.go
- internal/net/grpc/option.go
- internal/net/http/metrics/pprof.go
- internal/net/http/metrics/pprof_test.go
- internal/safety/safety.go
- k8s/agent/ngt/configmap.yaml
- k8s/discoverer/configmap.yaml
- k8s/discoverer/deployment.yaml
- k8s/gateway/gateway/lb/configmap.yaml
- k8s/gateway/gateway/lb/deployment.yaml
- k8s/manager/index/configmap.yaml
- k8s/manager/index/deployment.yaml
- k8s/operator/helm/crds/valdrelease.yaml
- pkg/gateway/lb/handler/grpc/search_benchmark_test.go
- rust/libs/proto/src/payload.v1.rs
- versions/GOLANGCILINT_VERSION
- versions/GO_VERSION
- versions/HELM_VERSION
- versions/JAEGER_OPERATOR_VERSION
- versions/K3S_VERSION
- versions/KUBECTL_VERSION
- versions/PROMETHEUS_STACK_VERSION
Additional Context Used
Learnings (1)
pkg/gateway/lb/handler/grpc/aggregation.go (4)
User: kpango PR: vdaas/vald#2491 File: pkg/gateway/lb/handler/grpc/aggregation.go:500-505 Timestamp: 2024-05-08T02:19:58.067Z Learning: User: kpango PR: vdaas/vald#2491 File: pkg/gateway/lb/handler/grpc/aggregation.go:500-505 Timestamp: 2024-05-08T02:17:42.078Z Learning: The `GetNum()` method in the `valdStdAggr` struct should include a nil check to prevent runtime panics when called on a nil instance. This is a defensive programming practice to enhance the robustness of the code.
User: kpango PR: vdaas/vald#2491 File: pkg/gateway/lb/handler/grpc/aggregation.go:500-505 Timestamp: 2024-05-08T02:17:42.078Z Learning: The `GetNum()` method in the `valdStdAggr` struct should include a nil check to prevent runtime panics when called on a nil instance. This is a defensive programming practice to enhance the robustness of the code.
User: kpango PR: vdaas/vald#2491 File: pkg/gateway/lb/handler/grpc/aggregation.go:689-690 Timestamp: 2024-05-07T03:58:42.344Z Learning: The direct slicing approach (`result = result[:v.fnum]`) is preferred over using `slices.Clip` for its simplicity and performance in the context of limiting the number of results in aggregation methods in `pkg/gateway/lb/handler/grpc/aggregation.go`.
User: kpango PR: vdaas/vald#2491 File: pkg/gateway/lb/handler/grpc/handler.go:306-316 Timestamp: 2024-05-07T04:33:44.775Z Learning: User: kpango PR: vdaas/vald#2491 File: pkg/gateway/lb/handler/grpc/handler.go:306-316 Timestamp: 2024-05-07T04:15:11.886Z Learning: In this project, vtprotobuf is used, which provides enhanced methods for protobuf objects: CloneVT (object cloning), EqualVT (object comparison), SizeVT (returns object size), MarshalVT (faster version of proto.Marshal), and UnmarshalVT (faster version of proto.Unmarshal). These methods offer better performance than usual protobuf usage.
Additional comments not posted (4)
apis/docs/v1/docs.md (1)
798-809
: The addition of theratio
field in the search configuration is well-documented. Ensure that all services using this configuration are updated to handle this new field.pkg/gateway/lb/handler/grpc/handler.go (3)
453-467
: Optimize the calculation logic incalculateNum
.
453-467
: Optimize the calculation logic incalculateNum
.
Line range hint
327-437
: Verify comprehensive error handling inSearchByID
.This function includes multiple layers of error handling and status updates. It's crucial to verify through unit tests that all potential error scenarios are covered effectively. Here's a script to check the coverage of error handling paths:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- Makefile.d/k3d.mk (1 hunks)
- Makefile.d/tools.mk (3 hunks)
Files skipped from review as they are similar to previous changes (2)
- Makefile.d/k3d.mk
- Makefile.d/tools.mk
9da530e
to
5cae30b
Compare
5cae30b
to
6dee899
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (3)
- Makefile.d/k3d.mk (1 hunks)
- Makefile.d/kind.mk (1 hunks)
- Makefile.d/tools.mk (3 hunks)
Files skipped from review as they are similar to previous changes (3)
- Makefile.d/k3d.mk
- Makefile.d/kind.mk
- Makefile.d/tools.mk
* add efficient search logic for large top-k users Signed-off-by: kpango <kpango@vdaas.org> * fix bug Signed-off-by: kpango <kpango@vdaas.org> --------- Signed-off-by: kpango <kpango@vdaas.org>
* fix: git add chart directory for release (#2356) Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * 🔖 🤖 Release v1.7.11 (#2360) (#2361) * 🔖 🤖 Release v1.7.11 * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in 578746e according to the output from Gofumpt and Prettier. Details: #2360 --------- Signed-off-by: Vdaas CI <vald@vdaas.org> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * Change docker scan timeout longer (#2363) * refactor code using golangci-lint (#2362) Signed-off-by: kpango <kpango@vdaas.org> * Create SECURITY.md (#2367) * Create SECURITY.md Signed-off-by: Yusuke Kato <kpango@vdaas.org> * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in e8609b0 according to the output from Gofumpt and Prettier. Details: #2367 --------- Signed-off-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * add commit hash build image (#2359) Signed-off-by: kpango <kpango@vdaas.org> * Make agent export index metrics to Pod k8s resource (#2319) * Add config * Add clusterRole and serviceAccount for agent * Add more metrics * Format * Add export index info duration configuration * Refactor internal/k8s * Move apply function to internal/k8s * Add more config * Remove unused import and variable * Use vald errors * Add pod name and pod namespace to hack e2e agent * Apply tagalign * Revert unnecesarry specification of PodName * Enable export index info to Kubernetes in agent templates only when the config is enabled * Fix agent clusterrole name * Update key name * Update comments * Add index count as export metrics * Ignore gomnd * Add index count entry to exportMetricsOnCreateIndex function * Update values schema * Fix formatter job * Remove unused defines * update docker build target platform selection rules (#2370) Signed-off-by: kpango <kpango@vdaas.org> Co-authored-by: Yusuke Kadowaki <yusuke.kadowaki.1231@gmail.com> * improve Backport Pull Request Title & Description Quality (#2373) Signed-off-by: kpango <kpango@vdaas.org> Co-authored-by: Kosuke Morimoto <ksk@vdaas.org> * Update workflow to release readreplica chart (#2383) * improve Backport Pull Request Title & Description Quality (#2373) Signed-off-by: kpango <kpango@vdaas.org> Co-authored-by: Kosuke Morimoto <ksk@vdaas.org> * Add readreplica chart release * update k8s/readreplica * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in 70f1dad according to the output from Gofumpt and Prettier. Details: #2383 * [bugfix] fix backport github value bypass failure (#2378) Signed-off-by: kpango <kpango@vdaas.org> * Revert backport.yaml --------- Signed-off-by: kpango <kpango@vdaas.org> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Kosuke Morimoto <ksk@vdaas.org> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * 💚 ♻️ Add Con-Bench helm chart to the Vald charts (#2388) Signed-off-by: vankichi <kyukawa315@gmail.com> * Delete unnecessary code for mirror (#2366) * fix: delete unnecessary code for mirror Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: make deps/install and format Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * improve Backport Pull Request Title & Description Quality (#2373) Signed-off-by: kpango <kpango@vdaas.org> Co-authored-by: Kosuke Morimoto <ksk@vdaas.org> * fix: naming of variable Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> --------- Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> Signed-off-by: kpango <kpango@vdaas.org> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Kosuke Morimoto <ksk@vdaas.org> * change JP logo to EN logo (#2369) Signed-off-by: kpango <kpango@vdaas.org> Co-authored-by: taisuou <pqrct237@gmail.com> * Add rotate-all option to rotator (#2305) * Add rotate-all option to rotator * Fix linter warnings * Make it possible to choose multiple rotation target * Add tests for parseReplicaID * Add tests for parseReplicaID * Hide testify in internal * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in dd84674 according to the output from Gofumpt and Prettier. Details: #2305 * Apply format * Lint * Lint --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> * Resolve kvs already closed before last saving (#2390) Signed-off-by: kpango <kpango@vdaas.org> * create continous benchmark doc (#2352) * improve Backport Pull Request Title & Description Quality (#2373) Signed-off-by: kpango <kpango@vdaas.org> Co-authored-by: Kosuke Morimoto <ksk@vdaas.org> * 📝 create continous benchmark doc Signed-off-by: vankichi <kyukawa315@gmail.com> * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in ace001a according to the output from Gofumpt and Prettier. Details: #2352 * ♻️ rafactor default params Signed-off-by: vankichi <kyukawa315@gmail.com> * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in 9880dfc according to the output from Gofumpt and Prettier. Details: #2352 * ♻️ Fix format Signed-off-by: vankichi <kyukawa315@gmail.com> * Revert "improve Backport Pull Request Title & Description Quality (#2373)" This reverts commit 4246def. * ♻️ remove dimension from con-bench Signed-off-by: vankichi <kyukawa315@gmail.com> * 📝 Add sample benchmark manifest Signed-off-by: vankichi <kyukawa315@gmail.com> * Update docs/performance/continuos-benchmatk.md Co-authored-by: Yusuke Kato <kpango@vdaas.org> Signed-off-by: Kiichiro YUKAWA <kyukawa315@gmail.com> * Update docs/performance/continuos-benchmatk.md Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Signed-off-by: Kiichiro YUKAWA <kyukawa315@gmail.com> --------- Signed-off-by: kpango <kpango@vdaas.org> Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: Kiichiro YUKAWA <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Kosuke Morimoto <ksk@vdaas.org> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * fix: build error of internal kvs test (#2396) Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * 🤖 Update license headers / Format Go codes and YAML files (#2397) Signed-off-by: Vdaas CI <vald@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * fix: disable protobuf dispatch for client (#2401) Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * update deps (#2404) Signed-off-by: kpango <kpango@vdaas.org> * 🔖 🤖 Release v1.7.12 (#2408) (#2409) * 🔖 🤖 Release v1.7.12 * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in 5bd8605 according to the output from Gofumpt and Prettier. Details: #2408 --------- Signed-off-by: Vdaas CI <vald@vdaas.org> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * Add tests for index information export (#2412) * Add tests for index information export * Add comments * 📝 Fix typo of file name (#2413) Signed-off-by: vankichi <kyukawa315@gmail.com> * Fix agent-faiss build failed (#2418) * 🐳 Fix agent-faiss build failed Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐳 Fix COPY binary Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐳 Change COPY binary (use ld-linux-x86-64) Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> * Fix the logic to determine docker image (#2410) * fix: update action to get tag list Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: refactor sub command execution Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: use tag list in detect ci container workflow Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: name for detected result Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: invalid variable bug Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: execution error and invalid variable Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: deleted unnecessary comment Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: deleted unnecessary parameters Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: use new determine tag logic in the docker build workflow Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: bugfix extra tags handling and small refactor Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: build tag rule or release branch based on tag list Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: typo Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: indent Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: refactor syntax Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: typo and invalid option error Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: use helm extra option Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: add comment for tag management Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: unsetbug of helm extra option Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: build target condition Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> --------- Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * Update build rule for nightly image (#2421) * fix: update build rule for nightly Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: indent and deleted unnecessary tag Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: update tag name when tag is unknown Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: add secondary tags for build performance Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: argument order Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> --------- Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * Add `index-operator` template implementation (#2375) * improve Backport Pull Request Title & Description Quality (#2373) Signed-off-by: kpango <kpango@vdaas.org> Co-authored-by: Kosuke Morimoto <ksk@vdaas.org> * [bugfix] fix backport github value bypass failure (#2378) Signed-off-by: kpango <kpango@vdaas.org> * [bugfix] fix backport github value bypass failure (#2381) Signed-off-by: kpango <kpango@vdaas.org> * Add index operator boilerplate * Add k8s controller * Add pod and job controllers * Add annotations in the pod reconciler * Update auto indexing service configurations * Update WithOnReconcileFunc signature to include context * Refactor debugging log * Refactor reconcile funcs * add nolint * remove unused parameters * lint * Add dockerfile and build ci * remove params * Add index-operator image build command to Makefile * Add index operator k8s templates * Update helm schema * Update readme * Add internal/k8s as a trigger for images that depends on controller-runtime * remove comment * Update pkg/index/operator/service/operator.go Co-authored-by: Yusuke Kato <kpango@vdaas.org> Signed-off-by: Yusuke Kadowaki <yusuke.kadowaki.1231@gmail.com> * Update pkg/index/operator/service/operator.go Co-authored-by: Yusuke Kato <kpango@vdaas.org> Signed-off-by: Yusuke Kadowaki <yusuke.kadowaki.1231@gmail.com> * Refactor * Add build commands * Refactor Co-authored-by: Yusuke Kato <kpango@vdaas.org> --------- Signed-off-by: kpango <kpango@vdaas.org> Signed-off-by: Yusuke Kadowaki <yusuke.kadowaki.1231@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Kosuke Morimoto <ksk@vdaas.org> * Fix output settings to determine-docker-image-tag action and release branch build tag name (#2423) * fix: add outut settings for action Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: tag name for release branch build Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: refactor Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> --------- Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: typo of execution rule (#2426) Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * Add flush API for deleting all vectors (#2273) * add flush api Signed-off-by: aknishid <koco155@gmail.com> * added flush api * fix typo * fix pb.go & fix conflict Signed-off-by: aknishid <koco155@gmail.com> * add flush api * add handler Flush Signed-off-by: aknishid <koco155@gmail.com> * add vald.Client flush * fix filter gateway Signed-off-by: aknishid <koco155@gmail.com> * fix handler cnts Signed-off-by: aknishid <koco155@gmail.com> * delete unused var Signed-off-by: aknishid <koco155@gmail.com> * add author Signed-off-by: Yamada Takuya <takuya.ymd@gmail.com> * fix error Signed-off-by: aknishid <koco155@gmail.com> * fix flush.go & ngt.go Signed-off-by: aknishid <koco155@gmail.com> * add flush to example Signed-off-by: aknishid <koco155@gmail.com> * fix gateway lb Signed-off-by: aknishid <koco155@gmail.com> * update flush proto Signed-off-by: aknishid <koco155@gmail.com> * fix docs api README.md Signed-off-by: aknishid <koco155@gmail.com> * Fix flush ctx Signed-off-by: aknishid <koco155@gmail.com> * update docs & comment Signed-off-by: aknishid <koco155@gmail.com> * after make format Signed-off-by: aknishid <koco155@gmail.com> * add e2e Signed-off-by: aknishid <koco155@gmail.com> * update example & tutorial Signed-off-by: aknishid <koco155@gmail.com> * Update internal/errors/ngt.go Co-authored-by: Kiichiro YUKAWA <kyukawa315@gmail.com> * fix review Signed-off-by: aknishid <koco155@gmail.com> * modify status.md Signed-off-by: aknishid <koco155@gmail.com> * add error handling Signed-off-by: aknishid <koco155@gmail.com> * update proto Signed-off-by: aknishid <koco155@gmail.com> * fix doc Signed-off-by: aknishid <koco155@gmail.com> * ♻️ Fix proto Signed-off-by: vankichi <kyukawa315@gmail.com> * use atomic bool Signed-off-by: aknishid <koco155@gmail.com> * fix proto Signed-off-by: aknishid <koco155@gmail.com> * update proto Signed-off-by: aknishid <koco155@gmail.com> * fix gateway error Signed-off-by: aknishid <koco155@gmail.com> * ✅ 🚧 Avoid e2e failing due to unimplemented handler and Add unimplmented handler method for the future Signed-off-by: vankichi <kyukawa315@gmail.com> * Update apis/proto/v1/payload/payload.proto Co-authored-by: Kiichiro YUKAWA <kyukawa315@gmail.com> Signed-off-by: aknishid <38955080+aknishid@users.noreply.github.com> --------- Signed-off-by: aknishid <koco155@gmail.com> Signed-off-by: Yamada Takuya <takuya.ymd@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: aknishid <38955080+aknishid@users.noreply.github.com> Co-authored-by: takuyaymd <takuya.ymd@gmail.com> Co-authored-by: takuyyam <takuyyam@yahoo-corp.jp> Co-authored-by: Kiichiro YUKAWA <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> * docs: add aknishid as a contributor for code, and maintenance (#2431) * docs: update README.md [skip ci] * docs: update .all-contributorsrc [skip ci] * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in abc4c52 according to the output from Gofumpt and Prettier. Details: #2431 --------- Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * update deps & add validation for Flush API when agent is Read Only (#2433) Signed-off-by: kpango <kpango@vdaas.org> * Update README.md (grammer) (#2440) Signed-off-by: Hrichik Mazumder <57860916+hrichiksite@users.noreply.github.com> * docs: add hrichiksite as a contributor for doc (#2441) * docs: update README.md [skip ci] * docs: update .all-contributorsrc [skip ci] * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in b8054d9 according to the output from Gofumpt and Prettier. Details: #2441 --------- Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * Added snapshot timestamp annotations to read replica agent (#2428) * Add lastTimeRotateReplicaTimestampAnnotationsKey constant * Use ptr package * Update annotations in ngt.go and rotator.go * Add labels to Pod struct and update annotations handling*** ***Introduce vald package for k8s related constants*** ***Implement rotateIfNeeded function to check if read replica rotation is needed * Refactor * set sample log level debug * Use k8s timestamp * Refactor * Format * Add MatchingLabels method to ValdK8sClientMock * Format * Fix test * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in 39b66fa according to the output from Gofumpt and Prettier. Details: #2428 * Rename constants Co-authored-by: Yusuke Kato <kpango@vdaas.org> * Refactor rotateIfNeeded function to use label selector for listing deployments * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in 6ace761 according to the output from Gofumpt and Prettier. Details: #2428 * FIx deepsource * Revert --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> * fix: bugfix version update for docker build (#2445) Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * Fix index job logic to pass DNS A record (#2438) * fix: choose user given target address in priority Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: deleted unnecessary code Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: add logic to connect the user given target addresses Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: add info log about new connection target Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: indent of configuration data Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: indent bug Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: deepsource warning Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> --------- Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * Fix operator-sdk version (#2447) * fix: down grade operator version Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: upgrade operator-sdk version Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix: operator-sdk version Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> --------- Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * add file name lint (#2417) Signed-off-by: kpango <kpango@vdaas.org> * fix: add extra option for ci-container build (#2451) Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * Add base of benchmark operator dashboard (#2430) * 📈 Add base of benchmark operator dashboard Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 Fix build benchmark images trigger Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 Add info tag for info metrics Signed-off-by: vankichi <kyukawa315@gmail.com> * 📈 showing job image and tags Signed-off-by: vankichi <kyukawa315@gmail.com> * 📈 add bench job status Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Update dashboard and rename Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Fix Signed-off-by: vankichi <kyukawa315@gmail.com> * Update internal/observability/metrics/tools/benchmark/benchmark.go Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Signed-off-by: Kiichiro YUKAWA <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: Kiichiro YUKAWA <kyukawa315@gmail.com> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * Implement index operator logic for read replica rotation (#2444) * Remove job reconciler * Add leader election to index operator * Add roles for leader election * Add operator logic for rotation * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in 844cca1 according to the output from Gofumpt and Prettier. Details: #2444 * Remove unnecesary change * Fix build * nits * Refactor * Fix clusterrole * Move to podv2 * Add job concurrency check * Refactor * Refactor * Fix heavy copying * Update helm schema * Fix package doc * Remove unused code and refactor For() method * Update target read replica ID in config * Update internal/k8s/podv2/option.go Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Signed-off-by: Yusuke Kadowaki <yusuke.kadowaki.1231@gmail.com> * Refactor * rename podv2 to v2/pod * Add client DI * Fix envkey helper * Set namespace from values.yaml * Use annotations to specify the id * Update schema * Update sample * Update values * Update schema * Fix spelling * Fix package name * Fix read replica e2e * Add job templates for rotation, creation, saving, and correction * Update config.go to decode k8s object * Update sample.yaml for index operator * Use job template from cfg * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in 584d1f0 according to the output from Gofumpt and Prettier. Details: #2444 * Fix tagalign * Fix config test * Fix config test * Disable read replica e2e for now --------- Signed-off-by: Yusuke Kadowaki <yusuke.kadowaki.1231@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * add inner product distance type for ngt (#2454) Signed-off-by: kpango <kpango@vdaas.org> * Fix e2e for read replica and add e2e for index operator (#2455) * Fix read replica e2e * Update default image list in action.yaml * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in 6cdac9f according to the output from Gofumpt and Prettier. Details: #2455 --------- Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * Add unit tests for index operator (#2460) * Add prototype tests * Refactor * Fix not to use kubeconfig * 🤖 Automatically update PULL_REQUEST_TEMPLATE and ISSUE_TEMPLATE (#2457) Signed-off-by: Vdaas CI <vald@vdaas.org> Co-authored-by: Kiichiro YUKAWA <kyukawa315@gmail.com> * Bugfix recreate benchmark job when operator reboot (#2463) * 🐛 Fix recreate benchmark job already runnnig when operator rebooted Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Fix benchmarkJobReconcile status handling Signed-off-by: vankichi <kyukawa315@gmail.com> * ✅ Fix test Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Update k8s dir Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Fix Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> * Refactor k8s types (#2462) * Refactor types * Refactor hdf5_test.go * Format * Refactor * add extra flags for NGT build phase (#2467) * Fix workflow trigger for backport pr creation (#2471) * Automatically add backport main label for release-pr (#2473) * feat: automatically add backport main label for release-pr Signed-off-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * fix: deleted unnecessary quotation Signed-off-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> --------- Signed-off-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * update deps (#2468) * update deps Signed-off-by: kpango <kpango@vdaas.org> * fix server_test.go due to the grpc-go updates Signed-off-by: kpango <kpango@vdaas.org> --------- Signed-off-by: kpango <kpango@vdaas.org> * Update README.md (#2479) Signed-off-by: Yusuke Kato <kpango@vdaas.org> * Implement client metrics interceptor for continuous benchmark job (#2477) * ♻️ Mount benchmark-operator-config to vald-benchmark-job Pod Signed-off-by: vankichi <kyukawa315@gmail.com> * 📈 add framework for send gRPC client metrics Signed-off-by: vankichi <kyukawa315@gmail.com> * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in a532b7d according to the output from Gofumpt and Prettier. Details: #2477 * 📈 Add client metric interceptor Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Change vbor chart for apply general job configuration Signed-off-by: vankichi <kyukawa315@gmail.com> * Update pkg/tools/benchmark/job/usecase/benchmarkd.go Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Signed-off-by: Kiichiro YUKAWA <kyukawa315@gmail.com> * 🐛 Fix build error and add comment Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Fix format Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: Kiichiro YUKAWA <kyukawa315@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * implement ngt-rs and its unit test and add GitHub Actions workflows (#2478) * implement test * update rust dependencies * fix * make format * add Github Actions workflows * fix * fix * fix * fix * fix * fix * fix * fix * fix --------- Co-authored-by: Yusuke Kato <kpango@vdaas.org> * 📈 Add client metrics panels for continuous benchmark job (#2481) Signed-off-by: vankichi <kyukawa315@gmail.com> * Add rust versions (#2482) * add rust versions * fix * add rust version * add rust-toolchain.toml * make format * Update continuous benchmark docs (#2485) * 📝 Update continous benchmark docs Signed-off-by: vankichi <kyukawa315@gmail.com> * style: format code with Gofumpt and Prettier This commit fixes the style issues introduced in e3ab1d2 according to the output from Gofumpt and Prettier. Details: #2485 * Update docs/performance/continuous-benchmark.md Signed-off-by: Kiichiro YUKAWA <kyukawa315@gmail.com> * 📝 Fix Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: Kiichiro YUKAWA <kyukawa315@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * update NGT version (#2484) * fix sed line (#2488) * 🤖 Automatically update PULL_REQUEST_TEMPLATE and ISSUE_TEMPLATE (#2487) Signed-off-by: Vdaas CI <vald@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * Add efficient search logic for large top-k users (#2491) * add efficient search logic for large top-k users Signed-off-by: kpango <kpango@vdaas.org> * fix bug Signed-off-by: kpango <kpango@vdaas.org> --------- Signed-off-by: kpango <kpango@vdaas.org> --------- Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> Signed-off-by: Vdaas CI <vald@vdaas.org> Signed-off-by: kpango <kpango@vdaas.org> Signed-off-by: Yusuke Kato <kpango@vdaas.org> Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: Kiichiro YUKAWA <kyukawa315@gmail.com> Signed-off-by: Yusuke Kadowaki <yusuke.kadowaki.1231@gmail.com> Signed-off-by: aknishid <koco155@gmail.com> Signed-off-by: Yamada Takuya <takuya.ymd@gmail.com> Signed-off-by: aknishid <38955080+aknishid@users.noreply.github.com> Signed-off-by: Hrichik Mazumder <57860916+hrichiksite@users.noreply.github.com> Signed-off-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: vdaas-ci <57942646+vdaas-ci@users.noreply.github.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> Co-authored-by: Yusuke Kadowaki <yusuke.kadowaki.1231@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Kosuke Morimoto <ksk@vdaas.org> Co-authored-by: Kiichiro YUKAWA <kyukawa315@gmail.com> Co-authored-by: taisho <taisuou@gmail.com> Co-authored-by: taisuou <pqrct237@gmail.com> Co-authored-by: aknishid <38955080+aknishid@users.noreply.github.com> Co-authored-by: takuyaymd <takuya.ymd@gmail.com> Co-authored-by: takuyyam <takuyyam@yahoo-corp.jp> Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com> Co-authored-by: Hrichik Mazumder <57860916+hrichiksite@users.noreply.github.com>
Description
In the current Vald search architecture, each agent receives and processes queries broadcasted by the load balancer (LB) exactly as they are received. For example, with a configuration of Top-K=10 and Agent Replica=5, the LB reduces the results from 50 (produced collectively by the agents) down to the top 10.
However, when the Top-K value is extremely high, such as 100,000, it is inefficient to search across all K multiplied by the number of replicas. To address this, we have developed a feature that adjusts the number of requests forwarded to the agents based on a configurable ratio.
This pull request introduces a 'Ratio' setting within the Search Config. This setting allows for the dynamic adjustment of the proportion of the Top-K results that are propagated to the agents. If the 'Ratio' is not configured or set to 1, the system will maintain the current behavior of forwarding 100% of the Top-K results. If set to 0.5, only 50% of the Top-K will be searched. Conversely, setting the Ratio to 1.2 will result in 120% of the Top-K being searched, effectively increasing the breadth of the search under certain conditions.
Related Issue
Versions
Checklist
Special notes for your reviewer
Summary by CodeRabbit
New Features
v1.30.0
.-fsS
and-fsSL
options across multiple Makefiles for improved security and reliability.curl
options.Bug Fixes
golangci-lint
,reviewdog
,kubectl
, andbuf
.Documentation
1.22.2
to1.22.3
.Refactor