-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🌱 Add distributed tracing framework #1211
Conversation
Functions to wrap a Client and add trace span annotations to objects. Based on OpenTracing, with a sample implementation for Jaeger. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Welcome @bboreham! |
Hi @bboreham. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@bboreham thank you for proposing/bringing open-telemetry integration to this library! i was looking for this for a period of time and also had a few PoC works before (in some of our internal projects) which is very close/similar to the current state of this pull. i think overall the proposed change is fairly intuitive but based on my humble experience from our practice, i think there're yet a few things not addressed in this pull:
and as far as i know this KEP was completely refactored to merely do the tracing for incoming requests at server side instead of controller frameworks, is it? the proposed change from this pull seems to be conflict w/ the latest KEP b/c having tracing at both server and client side can be be redundant i suppose.. cc @lavalamp |
Hi @yue9944882, lot of great points there. I'll respond in pieces:
Yes, I noticed that today (change was made 8 days ago).
Nit: it specified kube-apiserver, kube-scheduler, kube-controller-manager and kubelet, (most of?) which don't use the controller-runtime framework. But your point is a good one: I would now like to resurrect much of the text that was removed.
I think they are complementary. |
Yes, this is called out in the linked Google doc.
Great thought. I think for many specific cases this can be identified by a single controller.
Also true. I wonder if it can be fixed by adding something to Status? There isn't any handy bag-of-values field like Annotations, but maybe can be achieved as a Condition? Further, suppose the observed state changes; we'd expect a reconcillation which (given the above) would start a new span. |
cc @DirectXMan12 |
/ok-to-test |
@bboreham: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
(minor point of order: this would be a feature PR, not infra) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Neat so far, need to read through it a bit more detail later/catch up on the discussion.
One of the things I've also played around with in the past is hooking the event emitter into this pipeline as well, so we get k8s events showing up too.
pkg/tracing/logger.go
Outdated
|
||
func (t tracingLogger) Error(err error, msg string, keysAndValues ...interface{}) { | ||
t.Logger.Error(err, msg, keysAndValues...) | ||
t.Span.LogKV(append([]interface{}{"error", err, "message", msg}, keysAndValues...)...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there a way to avoid needing an extra allocation here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also, is there a semantic difference between WithTags
and the key-value pairs specified here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That LogKV()
call is through an interface so the memory will escape; it's unlikely to be possible to avoid an allocation. This particular line is only hit under error conditions so probably not performance-sensitive, but there are similar ones.
Tags apply to the whole span so it wouldn't be right to put details from one error as a tag, however the next line sets an error
tag on the span so you can pick it out as one that had at least one error.
I want to do that, but it looks awkward, since the existing code takes it from the controller, and there is no Context parameter to the event recorder interface. Unless you know a way? |
Code in this PR is now using OpenTelemetry instead of OpenTracing. One point of detail: span contexts inside objects now look like this:
which is a divergence from the (now-deleted) KEP I followed previously. |
/remove-lifecycle stale |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
Rotten issues close after 30d of inactivity. Send feedback to sig-contributor-experience at kubernetes/community. |
@k8s-triage-robot: Closed this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/reopen |
@dims: Reopened this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: bboreham The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Hi, I haven’t done much on this for a long time; do you need me to update it? |
@bboreham yes please! |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close |
@k8s-triage-robot: Closed this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/reopen |
@fstrudel: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/reopen Sorry I dropped this again. |
@bboreham: Reopened this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
func SetupJaeger(serviceName string) (io.Closer, error) { | ||
// Create and install Jaeger export pipeline | ||
flush, err := jaeger.InstallNewPipeline( | ||
jaeger.WithCollectorEndpoint("http://jaeger-agent.default:14268/api/traces"), // FIXME name? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we allow to specify the endpoint? (parameter/env variable/other?)
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close |
@k8s-triage-robot: Closed this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Is there a possibility that this PR will be merged? It appears to be the closest effort to adding tracing support in controller-runtime. |
Created in support of kubernetes-sigs/cluster-api#3760
This doc goes into more detail and is probably the best place to discuss.
What is in this PR:
client.Client
which will log all Kubernetes API calls to the current trace.logr.Logger
which will add all log messages to a trace.This PR does not create any tracing spans. You would use it in a controller like this (error handling elided):
The code is written using the OpenTracing API and Jaeger; I will look at porting it to OpenTelemetry.