-
Notifications
You must be signed in to change notification settings - Fork 487
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tail-based sampling #526
Tail-based sampling #526
Conversation
Our example Tempo Kubernetes manifests deploy it as a DaemonSet, and then uses a service to LB across the pods. Will this break that workflow? |
To be honest, I haven't tested it yet with the k8s examples, so I can't say for sure. I think it will. ATM I'm working on that, plus updating documentation and docker-compose examples. |
2275a23
to
9d541d5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall love this work and I think we should merge it shortly and begin experimenting with it.
Are the policies and load balancing separately configurable? So you can have one layer that does the load balancing and one that applies the policies?
pkg/tempo/config.go
Outdated
defaultDecisionWait = time.Second * 5 | ||
|
||
// defaultLoadBalancingPort is the default port the agent uses for internal load balancing | ||
defaultLoadBalancingPort = "9999" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe we should choose otlp default + 1? would give it a bit more meaning.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, 9999 was just a placeholder. Changed in 6ca6e3e
All right, my last commit 8d11011 adds tail-based sampling to the k3d example. It also introduces the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm. couple of clean up things
}, | ||
]), | ||
|
||
pvc: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pvc necessary for the load generator?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed in c25abf3, thanks
'--config=/etc/collector/config.yaml', | ||
), | ||
|
||
pvc: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no pvc necessary for otelcol
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed in c25abf3
Implements load balancing of spans by trace ID between agent instances for tail sampling
Many times users will have single instance deployements that do not require to load balance spans. Load balancing block is optional
* Add tail sampling in tempo pipelines * Add load balancing for traces when tail sampling Implements load balancing of spans by trace ID between agent instances for tail sampling * Lint things * Config recevier listening port * Add sampling in scraping service compose example * Don't load balance when it's not needed Many times users will have single instance deployements that do not require to load balance spans. Load balancing block is optional * Config reference for tail sampling * Update CHANGELOG * Fix panic * Move exporter to its own block * Make tail-sampling work in k3d example * Fix image name * Lower collector log level * Remove unnecessary pvcs
PR Description
Implement tail-based sampling. This is done by layering two pipelines: one to load balance spans across agent instances and another that does the actual tail sampling.
Which issue(s) this PR fixes
None
Notes to the Reviewer
PR is still WIP
PR Checklist