Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide interim sampling capability in exporter #5

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

garthk
Copy link
Owner

@garthk garthk commented May 20, 2020

There's plenty of progress on the OT specification. If you can wait, you don't need to use this capability. Reading material:

As the README change puts it:

Please treat this as an off-API workaround. We trust the OpenTelemetry specification will evolve to cover this use case, after which you'll be able to get this result without an implementation kludge.

That said, some Honeycomb customers need to be able to sample right now because of a change in their pricing model. It's literally a choice between sampling in a Honeycomb-sympathetic way, or turning it off. So, this branch provides a workaround:

  • You set some attribute on your spans eg. hc_sample to indicate your preferred sample rate, as a positive integer: 4 means you're dropping 75% of spans

  • You configure the exporter with samplerate_key set to something eg. "hc_sample"

  • The exporter pops "hc_sample" out of the attribute map so it's no longer confusable for a measurement, and _moves it to the HC samplerate key

  • The exporter also drops all but 1 in samplerate spans before export

It's crude, but it'll do for now.

@garthk garthk force-pushed the last-resort-sampling branch from 2679a50 to 128cc8f Compare May 20, 2020 11:17
@garthk garthk marked this pull request as ready for review May 20, 2020 11:18
@garthk garthk self-assigned this May 20, 2020
@garthk garthk changed the title WIP on sampling. FORK ME if you need to experiment. Provide interim sampling capability in exporter May 20, 2020
@garthk
Copy link
Owner Author

garthk commented May 23, 2020

open-telemetry/opentelemetry-specification#570 would provide an official span attribute eg.span.count with the number we need. I'll see if I can wire a sampler up as part of cleaning up and extending opentelemetry-beam/opentelemetry-examples#2.

Base automatically changed from trunk to main March 7, 2021 22:50
@aj-foster
Copy link

Hi @garthk. Could you update us on the current state of affairs of sampling with Honeycomb, and if workarounds like this continue to be necessary?

@garthk
Copy link
Owner Author

garthk commented Nov 10, 2022

👋 Honeycomb have since added native OpenTelemetry delivery, so you shouldn't need this exporter, for starters. Strikes me the correct approach now is to hook up processors to perform the sampling and tell Honeycomb your sample rate. According to their Python sampling , you can do it using environment variables with the Python SDK:

export OTEL_TRACES_SAMPLER="traceidratio"
export OTEL_TRACES_SAMPLER_ARG=0.5
export OTEL_RESOURCE_ATTRIBUTES="SampleRate=2"
# … and then run your software

… and those are in the OpenTelemetry Environment Variable Specification and also documented in the Erlang OTel docs.

For dynamic sampling, you'll want to set SampleRate as a span attribute rather than a resource. If you're adventurous, perhaps you can write your own Probability Sampling processor. I'm no longer working with Elixir, though, so I regret I can't give a hand.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants