Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus exporter produces invalid Prometheus text format #5693

Closed
lchdev opened this issue Aug 8, 2023 · 7 comments · Fixed by #6015
Closed

Prometheus exporter produces invalid Prometheus text format #5693

lchdev opened this issue Aug 8, 2023 · 7 comments · Fixed by #6015
Labels
Bug Something isn't working

Comments

@lchdev
Copy link

lchdev commented Aug 8, 2023

Describe the bug

The Prometheus exporter automatically adds target_info and otel_scope_info metadata metrics. Those metrics are gauges, but in their #TYPE line, the type is set to info, which is not a valid prometheus type.

As such, the export is not compliant to the spec and will be rejected by some scrappers.

What did you expect to see?

# TYPE target gauge
# HELP target Target metadata
target_info{*** SOME LABELS***} 1
# TYPE otel_scope_info gauge
# HELP otel_scope_info Scope metadata
otel_scope_info{*** SOME LABELS***} 1

What did you see instead?

# TYPE target info
# HELP target Target metadata
target_info{*** SOME LABELS***} 1
# TYPE otel_scope_info info
# HELP otel_scope_info Scope metadata
otel_scope_info{*** SOME LABELS***} 1

What version and what artifacts are you using?

Affects latest tag (1.28.0) and main branch.

@breedx-splk
Copy link
Contributor

Thanks @lchdev . It helps reviewers quite a bit if you can link to the spec when submitting issues. I think I managed to track it down, though. Is this the relevant section of the prom text spec? It looks like it to me.

@lchdev
Copy link
Author

lchdev commented Aug 8, 2023

@breedx-splk the spec you pointed is indeed the text based format, but the available metric types are listed here: https://prometheus.io/docs/concepts/metric_types/ (there are only 4 of them).

@dashpole
Copy link
Contributor

info is a valid type in OpenMetrics, but not in Prometheus. https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#info

@breedx-splk
Copy link
Contributor

I think this PR intended to point at this core repo and not the fork itself: lchdev#1

@lchdev
Copy link
Author

lchdev commented Aug 17, 2023

info is a valid type in OpenMetrics, but not in Prometheus. https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#info

Indeed, and as such, it should be converted when exposed by the Prometheus endpoint.

It's common practice to have "info" metrics exposed as gauged with a fixed value in Prometheus, so I proposed this simple change for those 2 hard-coded metrics in the Prometheus Serializer.

The same issue could however arise when trying to export metrics with types not defined in the Prometheus spec. I actually did not check the behavior in this case (I only have basic types such as gauges and counters to deal with). Choices would be to either convert them (if possible) or drop them ?

It's also worth noticing that the Prometheus server itself won't care about metric types, only some tooling / scrappers will.

I think this PR intended to point at this core repo and not the fork itself: lchdev#1

Yes indeed, I should have paid attention :/

I closed the original PR for now and will recreate it to upstream.

@jingyuanli001
Copy link

as a short term solution is there a way to skip/suppress these metrics somehow?

@Sharofiddin
Copy link

Please, can you provide a solution for until this branch published to release?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working
Projects
None yet
5 participants