Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

otlpgrpc: turn on round_robin LB policy and kuberesolver for resolving k8s service endpoints #5731

Merged
merged 5 commits into from
Jan 18, 2024

Conversation

yeya24
Copy link
Contributor

@yeya24 yeya24 commented Jan 17, 2024

What this PR does:

For tracing with opentelemetry, Cortex uses otlpgrpc protocol to send traces to OTLP backends.

When you have multiple tracing backends, it is always recommended to have load balancing. Since gRPC is a long lived connection, it is more recommended to do client side loadbalancing. For more details, can refer to https://github.com/grpc/grpc/blob/master/doc/load-balancing.md. In this pr, I added an option to enable round_robin policy, by default it is pick_first.

This pr also enables kuberesolver support which allows to resolve endpoint with prefix kubernetes://, for example kubernetes://opentelemetry-collector.tracing:4317 to be resolved as a k8s service and return the actual endpoint IPs (pod IP). This is helpful because kuberesolver watches the backend endpoints changes and allows client side loadbalancing to send requests to newly scaled up pods such as through HPA.

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

…er for kubernetes service endpoint

Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
@pull-request-size pull-request-size bot added size/M and removed size/S labels Jan 17, 2024
@yeya24 yeya24 merged commit ddcae19 into cortexproject:master Jan 18, 2024
14 checks passed
@yeya24 yeya24 deleted the clientside-grpc-lb branch January 18, 2024 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants