Record instrumentation_scope attribute in Prometheus export #2493
Assignees
Labels
spec:metrics
Related to the specification/metrics directory
Comments
|
@open-telemetry/wg-prometheus |
|
Related to #1906. |
|
Ah this is mostly a dupe of #1906 so feel free to close this if it makes sense. We need to prioritize it though, while we don't intend to publish Prometheus exporter as stable next month, with metrics going stable we expect a lot of bug reports for this since it's easy to run into. |
|
#2703 should close this. |
tigrannajaryan
pushed a commit
that referenced
this issue
Oct 12, 2022
Fixes: #2493 Related: #1906 This is a second attempt at #2422. ## Changes ### Background: Naming Collisions OpenTelemetry encourages the use of semantic conventions to make metric naming similar across instrumentation. For example, if I have two http client libraries in my application, they would each produce a metric named `http.client.duration`, but with different meters (e.g. [otelmux](https://github.com/open-telemetry/opentelemetry-go-contrib/tree/0dd27453a1ce8e433cb632e175a27f28ee83998d/instrumentation/github.com/gorilla/mux/otelmux) vs [otelhttp](https://github.com/open-telemetry/opentelemetry-go-contrib/tree/0dd27453a1ce8e433cb632e175a27f28ee83998d/instrumentation/net/http/otelhttp)). A prometheus exporter which receives both of these metrics would not be able to serve both of those histograms. This would occur anytime a user uses two libraries which produces the same category (e.g. http, database, rpc, etc) of metrics, or if the two libraries just happen to use the same name for a metric. Depending on the language, it may fail to create the Prometheus exporter, or may fail to send some, or all metrics if the same labels keys and values are present in both. ### Desired User Experience As a user, I can use a Prometheus exporter with OpenTelemetry without experiencing strange errors/behavior due to naming collisions, and without having to apply transformations to metric names to work around these, except in rare cases. As a user, I can easily add scope attributes to my metrics in Prometheus by joining with an info-style metric. This is a common pattern in Prometheus: https://grafana.com/blog/2021/08/04/how-to-use-promql-joins-for-more-effective-queries-of-prometheus-metrics-at-scale/. ### Design Add `opentelemetry_scope_name` and `opentelemetry_scope_version` as labels to all metrics. This ensures that if two libraries produce the same metric points, they don't collide because the scope name/version labels will differ. Those labels also serve as "join keys" to be able to add scope attributes to Prometheus metrics. This is accomplished by introducing an `opentelemetry_scope_info` metric containing the same `opentelemetry_scope_name` and `opentelemetry_scope_version` labels, but also including scope attributes. This also enables the collector's Prometheus receiver to reconstruct the original Instrumentation Scope when receiving the metrics.
carlosalberto
pushed a commit
to carlosalberto/opentelemetry-specification
that referenced
this issue
Oct 31, 2024
…elemetry#2703) Fixes: open-telemetry#2493 Related: open-telemetry#1906 This is a second attempt at open-telemetry#2422. ## Changes ### Background: Naming Collisions OpenTelemetry encourages the use of semantic conventions to make metric naming similar across instrumentation. For example, if I have two http client libraries in my application, they would each produce a metric named `http.client.duration`, but with different meters (e.g. [otelmux](https://github.com/open-telemetry/opentelemetry-go-contrib/tree/0dd27453a1ce8e433cb632e175a27f28ee83998d/instrumentation/github.com/gorilla/mux/otelmux) vs [otelhttp](https://github.com/open-telemetry/opentelemetry-go-contrib/tree/0dd27453a1ce8e433cb632e175a27f28ee83998d/instrumentation/net/http/otelhttp)). A prometheus exporter which receives both of these metrics would not be able to serve both of those histograms. This would occur anytime a user uses two libraries which produces the same category (e.g. http, database, rpc, etc) of metrics, or if the two libraries just happen to use the same name for a metric. Depending on the language, it may fail to create the Prometheus exporter, or may fail to send some, or all metrics if the same labels keys and values are present in both. ### Desired User Experience As a user, I can use a Prometheus exporter with OpenTelemetry without experiencing strange errors/behavior due to naming collisions, and without having to apply transformations to metric names to work around these, except in rare cases. As a user, I can easily add scope attributes to my metrics in Prometheus by joining with an info-style metric. This is a common pattern in Prometheus: https://grafana.com/blog/2021/08/04/how-to-use-promql-joins-for-more-effective-queries-of-prometheus-metrics-at-scale/. ### Design Add `opentelemetry_scope_name` and `opentelemetry_scope_version` as labels to all metrics. This ensures that if two libraries produce the same metric points, they don't collide because the scope name/version labels will differ. Those labels also serve as "join keys" to be able to add scope attributes to Prometheus metrics. This is accomplished by introducing an `opentelemetry_scope_info` metric containing the same `opentelemetry_scope_name` and `opentelemetry_scope_version` labels, but also including scope attributes. This also enables the collector's Prometheus receiver to reconstruct the original Instrumentation Scope when receiving the metrics.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently, instrumentation_scope is not used in the documentation for converting OTLP metrics to Prometheus. However, because it is part of identity for the OTLP model, it should be added as an attribute. Otherwise, the OTel SDK can record two metric streams with the same name and attributes, but different instrumentation scope, and the prometheus exporter will output them as identical streams causing a failure to scrape.
The text was updated successfully, but these errors were encountered: