Problems with prometheus on OTel 1.8

[dominikjeske]

Area

area:exporter:prometheus

Package Version

Package Name	Version
OpenTelemetry.Api	1.8.0
OpenTelemetry	1.8.0
OpenTelemetry.Exporter.OpenTelemetryProtocol	1.8.0
OpenTelemetry.Exporter.Prometheus.AspNetCore	.8.0-rc.1
OpenTelemetry.Exporter.Prometheus.HttpListener	1.8.0-rc.1
OpenTelemetry	1.8.0

Runtime Version

net8.0

Description

After update to new Otel 1.8 we are expirience issues in our prometheus. In one version it says "Error on ingesting samples that are too old or are too far into the future" and after cross check with new version of prometheus it says "Error on ingesting samples with different value but same timestamp". Additionally we observe that on some random queries to /metrics (we are getting otel_scope_info that is missing timestamp like described here prometheus/prometheus#6554). Like below

If I'm getting this right this info should be returned only when "Accept: application/openmetrics-text" is send but maybe I'm wrong. Additionally this info is random - one query return metrics with this info and another without.

If this behavior is by design please share some documentation.

ps. After downgrade to 1.7 we don't have those errors

Steps to Reproduce

Just use standard OTel and serve prometheus endpoint

Expected Result

Prometheus agent is grabbing data with success

Actual Result

Errors from description

Additional Context

No response

[WalissonPires]

Today I had the same problem after upgrading from version 1.6.0 to 1.8.1. (.NET 8)
I had to use version 1.7.0 as indicated by dominikjeske (thrx).

[reyang]

Duplicated with #5515. The bug was introduced by #5107 and fixed by #5623.

[dominikjeske]

@reyang Can we reopen this issue? Today I tried to use OTel 1.9 but still I have problems with metrics. Prometheus logs:

prometheus[603702]: ts=2024-06-26T13:18:31.847Z caller=scrape.go:1607 level=warn component="scrape manager" scrape_pool=webapi target=
http://xxxxxxxxx:8080/metrics
msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=1

When rollback to 1.7 everything looks good. Is this a bug or we have to use new prometheus to handle metrics from 1.8+?