Feat: Auto-label profiles based on request baggage

[bryanhuhta]

In order to support integrations with k6 load testing, we need to be able to pull key-value pairs out of the request baggage and create labels to add to the profile for that request. Instead of targeting specifically k6 baggage, I create a semi-extensible feature to attach arbitrary labels from the request baggage. Edit It made more sense to be opinionated about how we go from baggage to labels. The package no longer allows users to define arbitrary filter and transformation rules, rather we will only select baggage prefixed with k6. to be mapped to labels. This significantly simplified the API.

To use this feature in conjunction with k6, the LabelsFromBaggageHandler should wrap the routes that should have this auto-labeling behavior (ideally it wraps the http server mux to enable it for all handlers), like so:

http.Handle("/", otelhttp.NewHandler(http.HandlerFunc(index), "IndexHandler"))
http.Handle("/bike", otelhttp.NewHandler(http.HandlerFunc(bikeRoute), "BikeHandler"))
http.Handle("/scooter", otelhttp.NewHandler(http.HandlerFunc(scooterRoute), "ScooterHandler"))
http.Handle("/car", otelhttp.NewHandler(http.HandlerFunc(carRoute), "CarHandler"))

autoLabelHandler := k6.LabelsFromBaggageHandler(http.DefaultServeMux)
log.Fatal(http.ListenAndServe(":5000", autoLabelHandler))

This example adds auto-labeling to all the routes (/, /bike, /scooter, /car) for any request baggage with k6-specific members.

All of this code lives in the github.com/grafana/pyroscope-go/x/k6 module. x/k6 was specifically chosen because this API is experimental. We may chose to modify it or outright remove it at any time. Therefore, it made sense to place it in this package to signal the potential instability of the API. As we test this package in production environments and are satisfied with it, we can move it to contrib/k6 to make it an official, stable extension.

[CLAassistant]

All committers have signed the CLA.

[kolesnikovae]

Nice job! I've left a number of comments, but the main suggestion is to move the feature to a submodule (e.g., contrib/k6)

[kolesnikovae]

Another issue is that in Go opentracing we will override the goroutine labels: https://github.com/grafana/dskit/blob/01ae0599d043390ab206c46d4ab165e31c67d5ef/spanprofiler/tracer.go#L48-L52.

The opentracing API does not allow for passing context in a general case, therefore there is no a trivial solution to this (if we want to reuse the tracing instrumentation for our purposes). As a workaround, we could "re-inject" the goroutine labels, but that would require binding to runtime_getProfLabel, which is unsafe and may break at any time

[kolesnikovae]

What I would also like to learn more about and discuss in detail is the integration scenario as a whole because I definitely lack the context. I'm wondering if the use of pprof labels is required at all: my understanding is that load tests never run concurrently on the test subject system (I just can't imagine when it may make sense), therefore it might be sufficient to refer to the time range for a given test run (identified with the series labels, such as service_name + version). And if the user needs request-level profiling, it's probably better to use profiling and tracing integration instead

[kolesnikovae]

Should we go further and put k6 stuff into a submodule? contrib would be just a container for integration sub-modules.

Otherwise, once we add another integration, its dependencies would be added to the k6 package, and vice-versa

[bryanhuhta]

Yep! I agree with this move! I was attempting to do that last Friday but go work sync was fighting with me. It kept trying to update the go version of other modules.

I'll continue to work on this though and get it addressed before merging.

[bryanhuhta]

Okay, I got this fixed in https://github.com/grafana/pyroscope-go/pull/99/files/cbda95a5bdb83f75d53d811f4b921385e53e6ed0..016a1717a57831f63723fc46f4298531c17ac3f5. A couple things to note:

• I marked the k6 package as experimental by moving it to the x/k6 submodule because this API may change or be removed entirely at any time.
• I had to downgrade the go.opentelemetry.io/otel package from 1.27.0 to 1.17.0 to make the submodule compatible with go versions <1.21.

[bryanhuhta]

@kolesnikovae As you take another look, note that I moved the code into the x/k6 submodule to indicate this API is experimental. We can graduate it to contrib/k6 once we deem it stable.

I tested this code on a local instance of pyroscope and it works well:

We get all the k6 labels mapped correctly:

That said, we need to modify the upstream k6 js library to be more pragmatic about the k6.name baggage item, because right now it's mapping to URL and that results in significant cardinality.