[bugfix] Vald gRPC Client and Pool logic makes huge backoff #1953

kpango · 2023-02-14T12:11:31Z

Signed-off-by: kpango kpango@vdaas.org

Description:

This PR inclueds many features.

Delete unnecessary deepsource binary
Added multi CRUD E2E make command Update protobuf
Re-Construct MultiXXX APIs
- Fixes [Proposal] Refactor/Re-Construct MultiXXX APIs. #1940
- Remove gateway internal package due to the MultiXXX APIs change
- Add Multi Concurrency configuration to internal/config and charts
Update go module dependencies
Update go version
Update libs version
Added internal/slices package for fast sorting and replace sort to it.
- https://pkg.go.dev/golang.org/x/exp/slices
Refactor internal/client/v1/client/discoverer
Added grpc.WrapGRPCMethod function for additional information for backoff metrics
- https://github.com/vdaas/vald/blob/bugfix/lb-backoff/gateway-lb-exists-api-notfound-backoff-problem/internal/net/grpc/context.go#L23-L29
Refactor Backoff metrics key
Refactor internal/info package for more correct stack trace information
- https://github.com/vdaas/vald/blob/bugfix/lb-backoff/gateway-lb-exists-api-notfound-backoff-problem/internal/info/info.go#L290-L316
Update glg’s logd logic for more readable logging
- https://github.com/vdaas/vald/blob/bugfix/lb-backoff/gateway-lb-exists-api-notfound-backoff-problem/internal/log/glg/glg.go#L100-L123
Remove internal/net/grpc/addrs.go
- internal/net/grpc/client.go changes RoundRobin logic we don't need atomicAddrs management any more, so I deleted it.
Re-Construct internal/net/grpc/pool package for more efficient and correct connection pooling and health check
Refactor internal/net/grpc/clinet.go for more efficient and client management
Bugfix circuitbreaker metrics panic problem
- this bug restarts LBGW, I added deep-copy impl for metrics and it's works and no panic for head of pr-1953 impl
Added safety function wrap for unsafe goroutine execution
- errgroup.Do should be use with safety.RecoverFunc so I added it.
Fix index manager send on closed channel panic.

Related Issue:

close [Proposal] Refactor/Re-Construct MultiXXX APIs. #1940

Versions:

Go Version: 1.20
Docker Version: 20.10.8
Kubernetes Version: 1.22.0
NGT Version: 2.0.9

Checklist:

I have read the CONTRIBUTING document and completed our CLA agreement.
I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer:

cloudflare-workers-and-pages · 2023-02-14T12:11:53Z

Deploying with Cloudflare Pages

Latest commit:	`d091470`
Status:	✅ Deploy successful!
Preview URL:	https://c3ce081a.vald.pages.dev
Branch Preview URL:	https://bugfix-lb-backoff-gateway-lb.vald.pages.dev

View logs

vdaas-ci · 2023-02-14T12:12:07Z

[CHATOPS:HELP] ChatOps commands.

🙆‍♀️ /approve - approve
💌 /changelog - replace the PR body by changelog details
🍱 /format - format codes and add licenses
✅ /gen-test - generate test codes
🏷️ /label - add labels
⏪ /rebase - rebase main
🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

pkg/gateway/lb/handler/grpc/handler.go

codecov · 2023-02-14T12:37:16Z

Codecov Report

Patch coverage: 3.61% and project coverage change: +0.01 🎉

Comparison is base (0f6c1e6) 29.68% compared to head (d091470) 29.70%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1953      +/-   ##
==========================================
+ Coverage   29.68%   29.70%   +0.01%     
==========================================
  Files         366      364       -2     
  Lines       34101    34130      +29     
==========================================
+ Hits        10122    10137      +15     
- Misses      23573    23576       +3     
- Partials      406      417      +11

Impacted Files	Coverage Δ
hack/tools/metrics/main.go	`0.00% <0.00%> (ø)`
internal/circuitbreaker/manager.go	`0.00% <0.00%> (ø)`
internal/client/v1/client/agent/core/client.go	`0.00% <0.00%> (ø)`
internal/client/v1/client/discoverer/discover.go	`0.00% <0.00%> (ø)`
internal/client/v1/client/vald/option.go	`0.00% <ø> (ø)`
internal/client/v1/client/vald/vald.go	`0.00% <0.00%> (ø)`
internal/config/lb.go	`100.00% <ø> (ø)`
internal/errors/grpc.go	`75.00% <ø> (ø)`
internal/file/file.go	`12.00% <0.00%> (ø)`
internal/net/control/control.go	`98.09% <ø> (-0.02%)`	⬇️
... and 44 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

pkg/gateway/lb/handler/grpc/handler.go

sonatype-lift · 2023-02-15T09:42:26Z

internal/log/glg/glg.go

 }

 func (l *logger) Fatal(vals ...interface{}) {
-	l.glg.Fatal(vals...)
+	l.retry.Out(l.glg.Fatal, vals...)


typecheck: cannot use l.glg.Fatal (value of type func(val ...interface{})) as func(vals ...interface{}) error value in argument to l.retry.Out

❗❗ 2 similar findings have been found in this PR

🔎 Expand here to view all instances of this finding

File Path Line Number

internal/log/glg/glg.go 141

internal/log/glg/glg.go 145

Visit the Lift Web Console to find more details in your report.

ℹ️ Expand to see all @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Help us improve LIFT! (Sonatype LiftBot external survey)

Was this a good recommendation for you? _{Answering this survey will not impact your Lift settings.}

[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

github-actions · 2023-03-02T12:22:25Z

[FOSSA] The scan result will be available at https://app.fossa.com/projects/custom%2B21465%2Fvald/refs/branch/bugfix%2Flb-backoff%2Fgateway-lb-exists-api-notfound-backoff-problem/4555f6fd645d56a5c89f5b5b1c59d97d5e6ee048

vdaas-ci · 2023-03-02T12:31:32Z

Profile Report

type	vald-agent-ngt	vald-lb-gateway	vald-discoverer	vald-manager-index
cpu
heap

other images

Signed-off-by: kpango <kpango@vdaas.org>

github-actions · 2023-03-02T12:53:36Z

[FOSSA] The scan result will be available at https://app.fossa.com/projects/custom%2B21465%2Fvald/refs/branch/bugfix%2Flb-backoff%2Fgateway-lb-exists-api-notfound-backoff-problem/5326503793cac3bf0ff8521047955b8ad5469bfe

vdaas-ci · 2023-03-02T13:02:25Z

Profile Report

type	vald-agent-ngt	vald-lb-gateway	vald-discoverer	vald-manager-index
cpu
heap

other images

github-actions bot added area/gateway/lb area/internal size/M type/bug Something isn't working labels Feb 14, 2023

github-actions bot added the priority/medium label Feb 14, 2023

github-advanced-security bot found potential problems Feb 14, 2023

View reviewed changes

pkg/gateway/lb/handler/grpc/handler.go Fixed Show fixed Hide fixed

kpango force-pushed the bugfix/lb-backoff/gateway-lb-exists-api-notfound-backoff-problem branch 3 times, most recently from 85bad2e to 4c8b24d Compare February 14, 2023 12:35

github-actions bot added the size/L label Feb 14, 2023

github-actions bot removed the size/M label Feb 14, 2023

github-advanced-security bot found potential problems Feb 14, 2023

View reviewed changes

pkg/gateway/lb/handler/grpc/handler.go Fixed Show fixed Hide fixed

kpango force-pushed the bugfix/lb-backoff/gateway-lb-exists-api-notfound-backoff-problem branch from 4c8b24d to ac4ab69 Compare February 14, 2023 13:11

github-actions bot added team/sre SRE team type/ci labels Feb 14, 2023

kpango force-pushed the bugfix/lb-backoff/gateway-lb-exists-api-notfound-backoff-problem branch 5 times, most recently from d09391f to 38dc8b4 Compare February 15, 2023 09:27

sonatype-lift bot reviewed Feb 15, 2023

View reviewed changes

kpango force-pushed the bugfix/lb-backoff/gateway-lb-exists-api-notfound-backoff-problem branch from 38dc8b4 to d2112fe Compare February 15, 2023 14:09

github-actions bot added size/XL and removed size/L labels Feb 15, 2023

kpango force-pushed the bugfix/lb-backoff/gateway-lb-exists-api-notfound-backoff-problem branch 3 times, most recently from 70e66a2 to b8cd101 Compare February 16, 2023 08:55

kpango added actions/fossa actions/e2e-chaos actions/e2e-profiling actions/e2e-max-dim labels Mar 2, 2023

[bugfix] Vald gRPC Client and Pool logic makes huge backoff

d091470

Signed-off-by: kpango <kpango@vdaas.org>

kpango force-pushed the bugfix/lb-backoff/gateway-lb-exists-api-notfound-backoff-problem branch from 8a10538 to d091470 Compare March 2, 2023 12:38

github-actions bot added the size/XXL label Mar 2, 2023

kpango removed actions/e2e-deploy actions/fossa actions/e2e-chaos actions/e2e-profiling actions/e2e-max-dim labels Mar 2, 2023

github-actions bot removed the size/XXL label Mar 2, 2023

hlts2 approved these changes Mar 2, 2023

View reviewed changes

kpango added actions/e2e-deploy actions/fossa actions/e2e-chaos actions/e2e-profiling actions/e2e-max-dim labels Mar 2, 2023

vankichi approved these changes Mar 3, 2023

View reviewed changes

datelier approved these changes Mar 3, 2023

View reviewed changes

kevindiu approved these changes Mar 3, 2023

View reviewed changes

kpango merged commit 4a4673a into main Mar 3, 2023

kpango deleted the bugfix/lb-backoff/gateway-lb-exists-api-notfound-backoff-problem branch March 3, 2023 01:44

kpango mentioned this pull request Mar 3, 2023

[patch] Release v1.7.2 #1957

Merged

vankichi mentioned this pull request Mar 29, 2023

Bug fix: fix range concurrency branch rule #1986

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bugfix] Vald gRPC Client and Pool logic makes huge backoff #1953

[bugfix] Vald gRPC Client and Pool logic makes huge backoff #1953

kpango commented Feb 14, 2023 •

edited

Loading

cloudflare-workers-and-pages bot commented Feb 14, 2023 •

edited

Loading

vdaas-ci commented Feb 14, 2023

codecov bot commented Feb 14, 2023 •

edited

Loading

sonatype-lift bot Feb 15, 2023

github-actions bot commented Mar 2, 2023

vdaas-ci commented Mar 2, 2023

github-actions bot commented Mar 2, 2023

vdaas-ci commented Mar 2, 2023

Command	Usage
`@sonatype-lift ignore`	Leave out the above finding from this PR
`@sonatype-lift ignoreall`	Leave out all the existing findings from this PR
`@sonatype-lift exclude <file\|issue\|path\|tool>`	Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file

[bugfix] Vald gRPC Client and Pool logic makes huge backoff #1953

[bugfix] Vald gRPC Client and Pool logic makes huge backoff #1953

Conversation

kpango commented Feb 14, 2023 • edited Loading

Description:

Related Issue:

Versions:

Checklist:

Special notes for your reviewer:

cloudflare-workers-and-pages bot commented Feb 14, 2023 • edited Loading

Deploying with Cloudflare Pages

vdaas-ci commented Feb 14, 2023

codecov bot commented Feb 14, 2023 • edited Loading

Codecov Report

sonatype-lift bot Feb 15, 2023

Choose a reason for hiding this comment

github-actions bot commented Mar 2, 2023

vdaas-ci commented Mar 2, 2023

Profile Report

github-actions bot commented Mar 2, 2023

vdaas-ci commented Mar 2, 2023

Profile Report

kpango commented Feb 14, 2023 •

edited

Loading

cloudflare-workers-and-pages bot commented Feb 14, 2023 •

edited

Loading

codecov bot commented Feb 14, 2023 •

edited

Loading