Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add broken index backup #2034

Merged
merged 36 commits into from
Jun 7, 2023
Merged

add broken index backup #2034

merged 36 commits into from
Jun 7, 2023

Conversation

ykadowak
Copy link
Contributor

@ykadowak ykadowak commented May 22, 2023

Description:

The current behavior when loading broken indexes is to discard all of them and continue running with a clean state (remove continuous). This makes it impossible to investigate broken indexes later. To address this, this PR adds a feature to allow the configuration of the number of broken indexes to be saved, enabling the investigation of broken indexes later.

This PR also changes the path to store indexes when CoW is disabled, from n.path to n.path/origin to match with the path that's used when CoW is enabled.

  • implement ngt.rebuild to migrate for the backup process
  • implement migrate function to do the migration
  • added tests (basically TestNew covers everything)
  • added agent.ngt.broken_index_history_limit parameter
  • added internal/test/data/agent/ngt/validIndex/* as a test index data
  • added some VS Code settings to develop agent stand alone in the devcontainer

Related Issue:

Versions:

  • Go Version: 1.20.3
  • Docker Version: 20.10.8
  • Kubernetes Version: 1.22.0
  • NGT Version: 2.0.11

Checklist:

Special notes for your reviewer:

@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented May 22, 2023

Deploying with  Cloudflare Pages  Cloudflare Pages

Latest commit: f866a3f
Status: ✅  Deploy successful!
Preview URL: https://5865744b.vald.pages.dev
Branch Preview URL: https://feature-agent-broken-index-b.vald.pages.dev

View logs

@vdaas-ci
Copy link
Collaborator

[CHATOPS:HELP] ChatOps commands.

  • 🙆‍♀️ /approve - approve
  • 💌 /changelog - replace the PR body by changelog details
  • 🍱 /format - format codes and add licenses
  • /gen-test - generate test codes
  • 🏷️ /label - add labels
  • /rebase - rebase main
  • 🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

dockers/dev/Dockerfile Outdated Show resolved Hide resolved
dockers/dev/Dockerfile Outdated Show resolved Hide resolved
@ykadowak ykadowak changed the title [WIP] update devcontainer to run ngt locally [WIP] add broken index backup May 23, 2023
@codecov
Copy link

codecov bot commented May 23, 2023

Codecov Report

Patch coverage: 59.39% and project coverage change: +0.48 🎉

Comparison is base (e67a833) 29.50% compared to head (f866a3f) 29.99%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2034      +/-   ##
==========================================
+ Coverage   29.50%   29.99%   +0.48%     
==========================================
  Files         366      366              
  Lines       34664    34797     +133     
==========================================
+ Hits        10228    10437     +209     
+ Misses      24000    23879     -121     
- Partials      436      481      +45     
Impacted Files Coverage Δ
internal/config/ngt.go 100.00% <ø> (ø)
internal/core/algorithm/ngt/ngt.go 63.93% <ø> (+0.65%) ⬆️
internal/errors/errors.go 49.51% <0.00%> (-0.73%) ⬇️
internal/test/testdata.go 66.66% <ø> (ø)
pkg/agent/core/ngt/service/ngt.go 35.41% <60.49%> (+13.75%) ⬆️

... and 4 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@ykadowak ykadowak force-pushed the feature/agent/broken-index-backup branch from af53c44 to e6e014b Compare May 26, 2023 05:29
@github-actions github-actions bot added team/set SET team size/L and removed size/M labels May 26, 2023
@ykadowak ykadowak requested a review from kpango June 7, 2023 04:11
kpango
kpango previously approved these changes Jun 7, 2023
Copy link
Collaborator

@kpango kpango left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vdaas-ci
Copy link
Collaborator

vdaas-ci commented Jun 7, 2023

Profile Report

typevald-agent-ngtvald-lb-gatewayvald-discoverervald-manager-index
cpu
heap
other images

@vdaas-ci
Copy link
Collaborator

vdaas-ci commented Jun 7, 2023

Profile Report

typevald-agent-ngtvald-lb-gatewayvald-discoverervald-manager-index
cpu
heap
other images

@kpango kpango merged commit 96df080 into main Jun 7, 2023
@kpango kpango deleted the feature/agent/broken-index-backup branch June 7, 2023 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants