Add broken index count metrics #2083

ykadowak · 2023-06-21T04:37:15Z

Description:

This PR makes it possible to get the number of broken indexes as a metric.
To check it's working, deployed vald to the local k3d with the settings of dev-broken-index-backup.yaml, and confirmed that the corresponding metric is correctly displayed on Grafana with the settings of 01-vald-agent.yaml.

Related Issue:

Versions:

Go Version: 1.20.3
Docker Version: 20.10.8
Kubernetes Version: 1.22.0
NGT Version: 2.0.11

Checklist:

I have read the CONTRIBUTING document and completed our CLA agreement.
I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer:

vdaas-ci · 2023-06-21T04:37:33Z

[CHATOPS:HELP] ChatOps commands.

🙆‍♀️ /approve - approve
💌 /changelog - replace the PR body by changelog details
🍱 /format - format codes and add licenses
✅ /gen-test - generate test codes
🏷️ /label - add labels
⏪ /rebase - rebase main
🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

cloudflare-workers-and-pages · 2023-06-21T04:38:25Z

Deploying with Cloudflare Pages

Latest commit:	`2710d88`
Status:	✅ Deploy successful!
Preview URL:	https://4254651d.vald.pages.dev
Branch Preview URL:	https://feature-agent-broken-index-m.vald.pages.dev

View logs

codecov · 2023-06-21T05:15:11Z

Codecov Report

Patch coverage: 9.55% and project coverage change: -0.02 ⚠️

Comparison is base (85ad070) 29.94% compared to head (35ff84d) 29.92%.

❗ Current head 35ff84d differs from pull request most recent head 2710d88. Consider uploading reports for the commit 2710d88 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2083      +/-   ##
==========================================
- Coverage   29.94%   29.92%   -0.02%     
==========================================
  Files         369      369              
  Lines       35048    35083      +35     
==========================================
+ Hits        10496    10500       +4     
- Misses      24070    24099      +29     
- Partials      482      484       +2

Impacted Files	Coverage Δ
...ternal/observability/metrics/agent/core/ngt/ngt.go	`0.00% <0.00%> (ø)`
pkg/gateway/lb/handler/grpc/aggregation.go	`0.00% <0.00%> (ø)`
pkg/agent/core/ngt/service/ngt.go	`35.69% <68.00%> (+0.18%)`	⬆️

... and 3 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

kpango

LGTM

github-actions · 2023-06-26T03:48:10Z

docs/api/search.md

-      }
+    message Request {
+      repeated float vector = 1 [ (validate.rules).repeated .min_items = 2 ];
+      Config config = 2;


[LanguageTool] _{reported by reviewdog 🐶}
Possible typo: you repeated a word (ENGLISH_WORD_REPEAT_RULE)
Suggestions: Config
Rule: https://community.languagetool.org/rule/show/ENGLISH_WORD_REPEAT_RULE?lang=en-US
Category: MISC

github-actions · 2023-06-26T03:48:11Z

docs/api/search.md

-        Filter.Config ingress_filters = 6;
-        Filter.Config egress_filters = 7;
-      }
+    message Config {


[LanguageTool] _{reported by reviewdog 🐶}
This sentence does not start with an uppercase letter. (UPPERCASE_SENTENCE_START)
Suggestions: Message
URL: https://languagetool.org/insights/post/spelling-capital-letters/
Rule: https://community.languagetool.org/rule/show/UPPERCASE_SENTENCE_START?lang=en-US
Category: CASING

github-actions · 2023-06-26T03:48:11Z

docs/api/search.md

-      }
+    message Request {
+      repeated float vector = 1 [ (validate.rules).repeated .min_items = 2 ];
+      Config config = 2;


[LanguageTool] _{reported by reviewdog 🐶}
Possible typo: you repeated a word (ENGLISH_WORD_REPEAT_RULE)
Suggestions: Config
Rule: https://community.languagetool.org/rule/show/ENGLISH_WORD_REPEAT_RULE?lang=en-US
Category: MISC

github-actions · 2023-06-26T03:48:11Z

docs/api/search.md

-        Filter.Config ingress_filters = 6;
-        Filter.Config egress_filters = 7;
-      }
+    message Config {


[LanguageTool] _{reported by reviewdog 🐶}
This sentence does not start with an uppercase letter. (UPPERCASE_SENTENCE_START)
Suggestions: Message
URL: https://languagetool.org/insights/post/spelling-capital-letters/
Rule: https://community.languagetool.org/rule/show/UPPERCASE_SENTENCE_START?lang=en-US
Category: CASING

github-actions · 2023-06-26T03:48:11Z

docs/user-guides/client-api-config.md

+
+| Algorithm         | Description                                                             | Recommended Situation                                                                                                           |
+| :---------------- | :---------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------ |
+| `ConcurrentQueue` | Traditional Vald proprietary aggregation algorithm                      | Good performance when asynchronous aggregation from a large number of Agents and a huge number of threads.                      |


[LanguageTool] _{reported by reviewdog 🐶}
Specify a number, remove phrase, or simply use “many” or “numerous” (LARGE_NUMBER_OF[1])
Suggestions: many, numerous
URL: https://languagetool.org/insights/post/wordiness/
Rule: https://community.languagetool.org/rule/show/LARGE_NUMBER_OF?lang=en-US&subId=1
Category: REDUNDANCY

github-actions bot added area/agent/core area/internal team/set SET team size/XL type/feature New feature priority/medium labels Jun 21, 2023

github-actions bot added the area/helm label Jun 22, 2023

ykadowak changed the title ~~[WIP] Feature/agent/broken index metrics~~ Add broken index count metrics Jun 23, 2023

ykadowak requested review from a team, kevindiu and kpango and removed request for a team June 23, 2023 02:27

kevindiu previously approved these changes Jun 23, 2023

View reviewed changes

kpango previously approved these changes Jun 23, 2023

View reviewed changes

ykadowak dismissed stale reviews from kpango and kevindiu via 35ff84d June 26, 2023 03:46

github-actions bot added area/gateway/lb area/makefile team/sre SRE team type/ci size/XXL labels Jun 26, 2023

github-actions bot reviewed Jun 26, 2023

View reviewed changes

github-actions bot removed the size/XL label Jun 26, 2023

ykadowak force-pushed the feature/agent/broken-index-metrics branch from 35ff84d to 9245d7f Compare June 26, 2023 03:50

github-actions bot removed type/ci team/sre SRE team area/gateway/lb labels Jun 26, 2023

github-actions bot added size/XL and removed area/makefile size/XL labels Jun 26, 2023

ykadowak added the actions/e2e-deploy label Jun 26, 2023

ykadowak force-pushed the feature/agent/broken-index-metrics branch from 9245d7f to 2710d88 Compare June 26, 2023 05:02

ykadowak and others added 8 commits June 26, 2023 05:02

add metrics broken index count

925964d

refactor print

fb98aa8

update test for ngt_test.go

6554ecb

remove unused fmt

17aa0fb

style: Format code with prettier and gofumpt

1d5cc5a

add debug log

3fed8d6

add broken index count to grafana dashboard

ac824f7

add values for develop broken index pv backup locally

2710d88

github-actions bot added size/XL and removed size/XL labels Jun 26, 2023

vankichi requested review from kevindiu and kpango June 26, 2023 05:21

kpango approved these changes Jun 26, 2023

View reviewed changes

kevindiu approved these changes Jun 26, 2023

View reviewed changes

vankichi merged commit f1c432f into main Jun 26, 2023

vankichi deleted the feature/agent/broken-index-metrics branch June 26, 2023 05:43

hlts2 mentioned this pull request Sep 13, 2023

[patch] Release v1.7.7 #2180

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add broken index count metrics #2083

Add broken index count metrics #2083

ykadowak commented Jun 21, 2023 •

edited

Loading

vdaas-ci commented Jun 21, 2023

cloudflare-workers-and-pages bot commented Jun 21, 2023 •

edited

Loading

codecov bot commented Jun 21, 2023 •

edited

Loading

kpango left a comment

github-actions bot Jun 26, 2023

github-actions bot Jun 26, 2023

github-actions bot Jun 26, 2023

github-actions bot Jun 26, 2023

github-actions bot Jun 26, 2023

Add broken index count metrics #2083

Add broken index count metrics #2083

Conversation

ykadowak commented Jun 21, 2023 • edited Loading

Description:

Related Issue:

Versions:

Checklist:

Special notes for your reviewer:

vdaas-ci commented Jun 21, 2023

cloudflare-workers-and-pages bot commented Jun 21, 2023 • edited Loading

Deploying with Cloudflare Pages

codecov bot commented Jun 21, 2023 • edited Loading

Codecov Report

kpango left a comment

Choose a reason for hiding this comment

github-actions bot Jun 26, 2023

Choose a reason for hiding this comment

github-actions bot Jun 26, 2023

Choose a reason for hiding this comment

github-actions bot Jun 26, 2023

Choose a reason for hiding this comment

github-actions bot Jun 26, 2023

Choose a reason for hiding this comment

github-actions bot Jun 26, 2023

Choose a reason for hiding this comment

ykadowak commented Jun 21, 2023 •

edited

Loading

cloudflare-workers-and-pages bot commented Jun 21, 2023 •

edited

Loading

codecov bot commented Jun 21, 2023 •

edited

Loading