Add exemplar support to Prometheus exporter

[saul]

Fixes #3449
Design discussion issue - not found

Changes

Adds support for exporting histogram exemplars to OpenTelemetry.Exporter.Prometheus.AspNetCore and OpenTelemetry.Exporter.Prometheus.HttpListener. There are a couple of aspects that I'd appreciate some guidance on:

• The spec states "If no exemplars exist on a bucket, the highest exemplar from a lower bucket MUST be used, even though it is a duplicate of another bucket’s exemplar" but my implementation doesn't follow this. I'm wondering what the rationale is behind this as for large histograms I can imagine this would produce a lot of extraneous data.
• The spec states "For Prometheus pull endpoints, only a single exemplar is able to be added to each bucket, so the largest exemplar from each bucket MUST be used, if attaching exemplars" but exemplars as implemented in OTel.Net do NOT preserve the largest exemplar in each bucket on histograms. Simply the most recent value for that bucket is stored as the exemplar, regardless of whether it is larger than the current exemplar or not
• Do the exemplars need to be 'wiped' between scrapes of the /metrics endpoint? Or should the exemplars be persistent in that once a bucket has an exemplar, it will always be returned by the /metrics endpoint until it is replaced by another exemplar.
• Is this considered an API-breaking change? I wonder if it would be best to add a bool option to PrometheusAspNetCoreOptions to enable exemplars, with the default being disabled (i.e. current behaviour)
  • "Prometheus Exporters are not stable yet, so no concern about breaking changes here." @cijothomas

Merge requirement checklist

• CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
• Unit tests added/updated
• Appropriate CHANGELOG.md files updated for non-trivial changes
• Changes in public API reviewed (if applicable) - see question above re adding a PrometheusAspNetCoreOptions option

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: cijothomas / name: Cijo Thomas (4261b1d)
• ✅ login: saul / name: Saul Rennison (8550c31, 24b2c6a, 606b69e)
• ✅ login: Kielek / name: Piotr Kiełkowicz (bb24629)

[cijothomas]

Is this considered an API-breaking change? I wonder if it would be best to add a bool option to PrometheusAspNetCoreOptions to enable exemplars, with the default being disabled (i.e. current behaviour)

I think you can enable it by default, given Exemplars itself is off-by-default in the SDK! So if prometheus exporter is exporting exemplars, users has already made an explicit decision to enable it at the sdk.
Another thing - Prometheus Exporters are not stable yet, so no concern about breaking changes here.

[cijothomas]

• ❌ - login: @saul / name: Saul Rennison . The commit (fe78b00, a609c15) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

@saul Can you sign the CLA part, as it is a mandatory pre-req for accepting any contributions to OpenTelemetry. If you are working on behalf on an employer, you may want to consult your employer before CLA. (Note: This is a one time requirement only)

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[saul]

Hi @cijothomas the CLA has been signed and this is ready for review when you have time. Thanks

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[cijothomas]

@robertcoltheart Tagging you here, to see if you have some bandwidth to help review this PR, given you have made several contributions to the Prometheus Exporter!

[cijothomas]

The spec states "If no exemplars exist on a bucket, the highest exemplar from a lower bucket MUST be used, even though it is a duplicate of another bucket’s exemplar" but my implementation doesn't follow this. I'm wondering what the rationale is behind this as for large histograms I can imagine this would produce a lot of extraneous data.

I am not sure of the reasoning for this either...You can put a TODO in the code and state this is not implemented, and we can follow up with the spec.

[robertcoltheart]

From a technical perspective, this seems fine. I can't comment on whether this meets the Otel specification to a degree that is satisfactory, or whether there should be additional things added. Ie. I haven't done a mapping to see if there are any gaps between the spec and this PR.

[robertcoltheart]

Should this go into PrometheusSerializer.cs alongside the rest of the byte-writing code for metrics?

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.