Prometheus Receiver translates metrics to OTLP metrics with non-cumulative temporality

[huyan0]

I am doing an end-to-end test for the Prometheus remote write exporter in PR #1525, and I am using the Prometheus receiver to get input metrics.

In the process, I discovered counter, histogram, and summary Prometheus metric scraped by the Prometheus receiver are not exportable by the remote write exporter, but gauge metrics are exported. This is because the remote write exporter drops non-cumulative monotonic int64, monotonic double, histogram, and summary, but doesn’t check the temporality on int64/double OTLP metrics.

The Prometheus receiver seems to translate counter, histogram, and summary metrics into OTLP metrics with non-cumulative temporality. Is this the expected behavior or should the Prometheus receiver/translation gets modified?

If this is the expected behavior, what’s an alternate receiver to use to ensure cumulative metrics of different types will not be dropped by the exporter?

cc @huyan0 @alolita @jmacd @bogdandrutu

[huyan0]

I replaced Prometheus receiver with a OTLP receiver; the metrics remained cumulative and was exported successfully.

[djjayeeta]

I have the same issue but I am using prometheus receiver with file exporter, I only see gauge metrics. My metrics are exposed in prometheus format, so I am left with the choice of using prometheus receiver.
Is this an expected behavior?
Which exporters works with histogram metrics + prometheus receiver?

[huyan0]

An update on this issue. This is the log I am getting from the Prometheus remote write exporter:

2020/08/27 16:44:34 invliad type and temp
2020/08/27 16:44:34 name:"demo_my_counter_metric" description:"The description of my_counter_metric" type:MONOTONIC_DOUBLE 
2020/08/27 16:44:34 temporality:  INVALID_TEMPORALITY

This shows the OC to OT translation is setting the temporality to INVALID_TEMPORALITY(not setting it at all perhaps?); this makes the metrics not pass the check of the Prometheus remote write exporter.

[huyan0]

With OTLP v0.5 temporality translation is added. The Prometheus receiver works with the remote write exporter