EMF: Store the initial value for cumulative metrics

[bjrara]

Why do we need it?
This is a follow-up PR of open-telemetry/opentelemetry-collector#3047.

In this PR, EMF exporter changes its strategy when handling cumulative metric values such as counters, and prometheus summaries - it stores the first scraped records as initial value in cache, and skip sending them to the backend. As a result, the deltas calculated from the second are treated as the first values that EMF exporter will export.

For example, if counter metrics are received in the following order:

1st batch: request_total {code=400, value=100}
2nd batch: request_total {code=400, value=101}
3rd batch: request_total {code=400, value=102}

The metrics will be sent by EMF as:
1st batch: SKIP
2nd batch: request_total {code=400, value=1}
3rd batch: request_total {code=400, value=2}

1st batch: SKIP
2nd batch: request_total {code=400, value=1}
3rd batch: request_total {code=400, value=1}

[Aneurysm9]

I'm having trouble understanding why this is necessary. If the exporter needs to translate cumulative metrics to deltas, shouldn't the initial sample still be present? I'd expect the delta output stream to look something like this:

1st batch: request_total {code=400, value=100}
2nd batch: request_total {code=400, value=1}
3rd batch: request_total {code=400, value=2}

The alternative means that I can never know the actual magnitude of the tracked value since the baseline is missing.

Can some more prose be added to the README explaining what this adjustment process is doing and why?

[bjrara]

I'm having trouble understanding why this is necessary. If the exporter needs to translate cumulative metrics to deltas, shouldn't the initial sample still be present? I'd expect the delta output stream to look something like this:

1st batch: request_total {code=400, value=100}
2nd batch: request_total {code=400, value=1}
3rd batch: request_total {code=400, value=2}

The alternative means that I can never know the actual magnitude of the tracked value since the baseline is missing.

Can some more prose be added to the README explaining what this adjustment process is doing and why?

Just realized the example in the description was wrong..., I corrected the adjusted value in EMF part. Sorry about the confusion.

Not sure if the issue becomes clearer as the example is corrected. In EMF exporter, we send deltas (of the current value and its previous value) to the CloudWatch backend when handling cumulative metrics.
If we scrape metrics every one minute, deltas will be the amount that is increased in that one minute.

Take the case above as an example, as request_total is increased by 1 every minute, delta in the 2nd, 3rd minute should be 1.

[bjrara]

I'm having trouble understanding why this is necessary. If the exporter needs to translate cumulative metrics to deltas, shouldn't the initial sample still be present? I'd expect the delta output stream to look something like this:

1st batch: request_total {code=400, value=100}
2nd batch: request_total {code=400, value=1}
3rd batch: request_total {code=400, value=2}

The alternative means that I can never know the actual magnitude of the tracked value since the baseline is missing.

Can some more prose be added to the README explaining what this adjustment process is doing and why?

IMHO, as we are exporting deltas by the most recent two records, we don't need the baseline, we only need know what its previous metric value is. The delta concept is alike rate() in Prometheus, but instead of user defining [5m] in the expression, the time window is decided by the scrape interval.

Previously, prometheus receiver sent the delta of the current metric value and the initial one, so EMF exporter could take it for granted that the first record received is delta-ed (by its previous value which is also the initial), for instance:

Metrics received from prometheus receiver were:

1st batch: SKIP (EMF wouldn't receive it)
2nd batch: request_total {code=400, value=1}
3rd batch: request_total {code=400, value=2}

And EMF exporter would send them as:

1st batch: NOT RECEIVED
2nd batch: request_total {code=400, value=1}
3rd batch: request_total {code=400, value=1}

Now since the logic in Prometheus receiver is changed, the cache and skip for the initials needs to be handled in EMF exporter instead.

Metrics received from prometheus receiver are:

1st batch: request_total {code=400, value=100}
2nd batch: request_total {code=400, value=101}
3rd batch: request_total {code=400, value=102}

And EMF exporter will send them as:

1st batch: SAVE as initial and SKIP
2nd batch: request_total {code=400, value=1}
3rd batch: request_total {code=400, value=1}

[mxiamxia]

LGTM

[bogdandrutu]

Please rebase to fix conflicts

[bjrara]

Please rebase to fix conflicts

Done. Thx!

[mircohacker]

Hey @bjrara @Aneurysm9 @mxiamxia this behaviour does not work in my usecase. Is there a way to disable it or a work around?