Support for OpenMetrics standard

[sirainen]

Feature Request

Telegraf supports Prometheus format, which was used as the basis for the OpenMetrics standard. However, there are some differences. Most importantly from my point of view, Dovecot v2.3.14+ uses the "info" type, which Telegraf treats as invalid. It would be nice if Telegraf would add support for this to make it more OpenMetrics standard compliant. (This can't be fixed on Dovecot side without making it violate the OpenMetrics standard, or adding a new configuration option to choose the behavior.)

See https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md

[powersj]

next steps: look into creating a new parser using the existing Prometheus parser as a foundation to correctly parse the new features like info.

[jadbaz]

Hello,
I have an upcoming project where I'm interested in using telegraf with OpenMetrics.
I'm wondering if this is on the roadmap for any time soon
Thank you

[1tft]

We have also issues with consuming different OpenMetrics endpoints. Have trouble with timestamp format.
Currently [[inputs.prometheus]] only accepts milliseconds timestamp (prometheus format specification).

OpenMetrics describes another timestamp format: Timestamps MUST be Unix Epoch in seconds.
You can use milli- or nanoseconds precision with a float64 like: 1520879607.123 or 1520879607.123456

Maybe it's possible to add also this timestamp format to [[inputs.prometheus]], so that we can already consume metrics from OpenMetrics endpoints till telegraf officially supports OpenMetrics format?

[1tft]

So we identified following main differences/evolution to prometheus format:

• new Info and StateSet type
• timestamp format (Unix Epoch in seconds. You can use milli- or nanoseconds precision with a float64 like: 1520879607.123 or 1520879607.123456)

Short summary OpenMetrics vs. Prometheus format: https://www.robustperception.io/openmetrics-is-released/

[yzhuang93]

While this is still in progress, is there any workaround that allows users to exclude or skip those OpenMetrics types like 'Info' and 'StateSet'? We are currently facing this issue in the usecase that opentelemetry java agent's prometheus exporter exports the metrics with 'Info' type but telegraf will just fail the whole parsing.

[srebhan]

@sirainen, @1tft and @yzhuang93 please test the binary in PR #15298, available as soon as CI finished the tests, and let me know if it works for you. Please be aware that by default inputs.prometheus uses the Content-Type header provided by the endpoint to decide if parsing should be done using the prometheus or openmetrics parser. In case your endpoint provides the wrong content-type please use the content_type_override setting in the prometheus input.

[1tft]

Started some tests on base of script output (inputs.exec) or file input (inputs.file) and openmetrics parser looks really good.

[yzhuang93]

@srebhan appreciate for the fix, i tried the build from your PR today as well, unfortunately, in my case when I specify the content_type_override to openmetrics-text, I got this error:

parsing text format failed: parsing failed: data does not end with # EOF

If I change the content_type_override to text, then it bypass the EOF issue but still not able to parse the openmetrics (of course), meanwhile, let me read your PR a bit to understand more on this.

[powersj]

Per the OpenMetrics spec, messages must end with EOF:

Expositions MUST end with EOF and SHOULD end with 'EOF\n'.

edit: opentelemetry -> openmetrics

[yzhuang93]

I see, then somehow the opentelemetry java agent is not exporting that to the output, let me double check the version I am using and retry with this build, thanks

[srebhan]

@yzhuang93 without the EOF, this is probably a prometheus message?

[yzhuang93]

@srebhan Yeah I think it is a Prometheus text format, it is exported by the https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#prometheus-exporter, but what confuses me is that it does contain the info type which is the OpenMetrics type

# TYPE target info
# HELP target Target metadata
...

And if I use the Prometheus instead of telegraf to scrape the data, it does the parsing without issue.

[srebhan]

@yzhuang93 ok, so you should go with the prometheus spec then.... It also might be OpenMetrics 0.0.4 but this parser does not support that. It is however the same format as the Prometheus text format I believe with a few additional types...

[yzhuang93]

Got it, thanks for the info @srebhan

[srebhan]

@1tft just to be sure, are you fine with the PR or do you need more testing?

[1tft]

@srebhan Yes. But had no chance to test with an endpoint of a standard software product.
Currently our use cases are only parsing OpenMetrics from txt-files and stdout of scripts (via exec). This is working fine. Thank you so much.

[srebhan]

I'll mark the PR as ready then. Please report any issues once you extend your use-case!