Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metrics spec supplementary guidelines #1966

Merged
merged 19 commits into from
Sep 30, 2021
218 changes: 218 additions & 0 deletions specification/metrics/supplementary-guidelines.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
# Supplementary Guidelines

Note: this document is NOT a spec, it is provided to support the Metrics
[API](./api.md) and [SDK](./sdk.md) specifications, it does NOT add any extra
requirements to the existing specifications.

Table of Contents:

* [Guidelines for instrumentation library
authors](#guidelines-for-instrumentation-library-authors)
* [Guidelines for SDK authors](#guidelines-for-sdk-authors)
* [Memory efficiency](#memory-efficiency)

## Guidelines for instrumentation library authors

TBD

## Guidelines for SDK authors

### Memory efficiency

The OpenTelemetry Metrics [Data Model](./datamodel.md) and [SDK](./sdk.md) are
designed to support both Cumulative and Delta
[Temporality](./datamodel.md#temporality). It is important to understand that
temporality will impact how the SDK could manage memory usage. Let's take the
following HTTP requests example:

* During the time range (T<sub>0</sub>, T<sub>1</sub>]:
* verb = `GET`, status = `200`, duration = `50 (ms)`
* verb = `GET`, status = `200`, duration = `100 (ms)`
* verb = `GET`, status = `500`, duration = `1 (ms)`
* During the time range (T<sub>1</sub>, T<sub>2</sub>]:
* no HTTP request has been received
* During the time range (T<sub>2</sub>, T<sub>3</sub>]
* verb = `GET`, status = `500`, duration = `5 (ms)`
* verb = `GET`, status = `500`, duration = `2 (ms)`
* During the time range (T<sub>3</sub>, T<sub>4</sub>]:
* verb = `GET`, status = `200`, duration = `100 (ms)`
* During the time range (T<sub>4</sub>, T<sub>5</sub>]:
* verb = `GET`, status = `200`, duration = `100 (ms)`
* verb = `GET`, status = `200`, duration = `30 (ms)`
* verb = `GET`, status = `200`, duration = `50 (ms)`

Let's imagine we export the metrics as [Histogram](./datamodel.md#histogram),
and to simplify the story we will only have one histogram bucket `(-Inf, +Inf)`:

If we export the metrics using **Delta Temporality**:

* (T<sub>0</sub>, T<sub>1</sub>]
* dimensions: {verb = `GET`, status = `200`}, count: `2`, min: `50 (ms)`, max:
reyang marked this conversation as resolved.
Show resolved Hide resolved
`100 (ms)`
* dimensions: {verb = `GET`, status = `500`}, count: `1`, min: `1 (ms)`, max:
`1 (ms)`
* (T<sub>1</sub>, T<sub>2</sub>]
* nothing since we don't have any Measurement received
* (T<sub>2</sub>, T<sub>3</sub>]
* dimensions: {verb = `GET`, status = `500`}, count: `2`, min: `2 (ms)`, max:
`5 (ms)`
* (T<sub>3</sub>, T<sub>4</sub>]
* dimensions: {verb = `GET`, status = `200`}, count: `1`, min: `100 (ms)`,
max: `100 (ms)`
* (T<sub>4</sub>, T<sub>5</sub>]
* dimensions: {verb = `GET`, status = `200`}, count: `3`, min: `30 (ms)`, max:
`100 (ms)`

You can see that the SDK **only needs to track what has happened after the
latest collection/export cycle**. For example, when the SDK started to process
measurements in (T<sub>1</sub>, T<sub>2</sub>], it can completely forget about
what has happened during (T<sub>0</sub>, T<sub>1</sub>].

If we export the metrics using **Cumulative Temporality**:

* (T<sub>0</sub>, T<sub>1</sub>]
* dimensions: {verb = `GET`, status = `200`}, count: `2`, min: `50 (ms)`, max:
`100 (ms)`
* dimensions: {verb = `GET`, status = `500`}, count: `1`, min: `1 (ms)`, max:
`1 (ms)`
* (T<sub>0</sub>, T<sub>2</sub>]
* dimensions: {verb = `GET`, status = `200`}, count: `2`, min: `50 (ms)`, max:
`100 (ms)`
* dimensions: {verb = `GET`, status = `500`}, count: `1`, min: `1 (ms)`, max:
`1 (ms)`
* (T<sub>0</sub>, T<sub>3</sub>]
* dimensions: {verb = `GET`, status = `200`}, count: `2`, min: `50 (ms)`, max:
`100 (ms)`
* dimensions: {verb = `GET`, status = `500`}, count: `3`, min: `1 (ms)`, max:
`5 (ms)`
* (T<sub>0</sub>, T<sub>4</sub>]
* dimensions: {verb = `GET`, status = `200`}, count: `3`, min: `50 (ms)`, max:
`100 (ms)`
* dimensions: {verb = `GET`, status = `500`}, count: `3`, min: `1 (ms)`, max:
`5 (ms)`
* (T<sub>0</sub>, T<sub>5</sub>]
* dimensions: {verb = `GET`, status = `200`}, count: `6`, min: `30 (ms)`, max:
`100 (ms)`
* dimensions: {verb = `GET`, status = `500`}, count: `3`, min: `1 (ms)`, max:
`5 (ms)`

If we choose **Cumulative Temporality**, the SDK **has to track what has
happened prior to the latest collection/export cycle**, in this worst case, the
SDK **will have to remember what has happened since the ever beginning of the
process**. This is known as Delta->Cumulative conversion.

Imagine if we have a long running service and we collect metrics with 7
dimensions, and each dimension can have 30 different values. We might eventually
end up having to remember the complete set of all `21,870,000,000` permutations!
This **cardinality explosion** a well-known challenge in the metrics space.

Making it even worse, if we export the permutations even if there are no recent
updates, the export batch could become huge and will be very costly. For
example, do we really need/want to export the same thing for (T<sub>0</sub>,
T<sub>2</sub>] in the above case?

So here are some suggestions that we encourage SDK implementers to consider:

* You want to control the memory usage rather than allow it to grow indefinitely
/ unbounded - regardless of what aggregation temporality is being used.
* You want to improve the memory efficiency by being able to **forget about
things that are no longer needed**.
* You probably don't want to keep exporting the same thing over and over again,
if there is no updates. You might want to consider [Resets and
Gaps](./datamodel.md#resets-and-gaps). For example, if a Cumulative metrics
stream hasn't received any updates for a long period of time, would it be okay
to reset the start time?

Now we can explore a more interesting topic, Cumulative->Delta conversion.

In the above case, we have Measurements reported by a [Histogram
Instrument](./api.md#histogram). What if we collect measurements from an
[Asynchronous Counter](./api.md#asynchronous-counter)?

The following example shows the number of [page
faults](https://en.wikipedia.org/wiki/Page_fault) of each thread since the
thread ever started:

* During the time range (T<sub>0</sub>, T<sub>1</sub>]:
* pid = `1001`, tid = `1`, #PF = `50`
* pid = `1001`, tid = `2`, #PF = `30`
* During the time range (T<sub>1</sub>, T<sub>2</sub>]:
* pid = `1001`, tid = `1`, #PF = `53`
* pid = `1001`, tid = `2`, #PF = `38`
* During the time range (T<sub>2</sub>, T<sub>3</sub>]
* pid = `1001`, tid = `1`, #PF = `56`
* pid = `1001`, tid = `2`, #PF = `42`
* During the time range (T<sub>3</sub>, T<sub>4</sub>]:
* pid = `1001`, tid = `1`, #PF = `60`
* pid = `1001`, tid = `2`, #PF = `47`
* During the time range (T<sub>4</sub>, T<sub>5</sub>]:
* thread 1 died, thread 3 started
* pid = `1001`, tid = `2`, #PF = `53`
* pid = `1001`, tid = `3`, #PF = `5`

If we export the metrics using **Cumulative Temporality**:

* (T<sub>0</sub>, T<sub>1</sub>]
* dimensions: {pid = `1001`, tid = `1`}, sum: `50`
* dimensions: {pid = `1001`, tid = `2`}, sum: `30`
* (T<sub>0</sub>, T<sub>2</sub>]
* dimensions: {pid = `1001`, tid = `1`}, sum: `53`
* dimensions: {pid = `1001`, tid = `2`}, sum: `38`
* (T<sub>0</sub>, T<sub>3</sub>]
* dimensions: {pid = `1001`, tid = `1`}, sum: `56`
* dimensions: {pid = `1001`, tid = `2`}, sum: `42`
* (T<sub>0</sub>, T<sub>4</sub>]
* dimensions: {pid = `1001`, tid = `1`}, sum: `60`
* dimensions: {pid = `1001`, tid = `2`}, sum: `47`
* (T<sub>0</sub>, T<sub>5</sub>]
* dimensions: {pid = `1001`, tid = `2`}, sum: `53`
* dimensions: {pid = `1001`, tid = `3`}, sum: `5`

It is quite straightforward - we just take the data being reported from the
asynchronous instruments and send them. We might want to consider if [Resets and
Gaps](./datamodel.md#resets-and-gaps) should be used to denote the end of a
metric stream - e.g. thread 1 died, the thread ID might be reused by the
operating system, and we probably don't want to confuse the metrics backend.

If we export the metrics using **Delta Temporality**:

* (T<sub>0</sub>, T<sub>1</sub>]
* dimensions: {pid = `1001`, tid = `1`}, delta: `50`
* dimensions: {pid = `1001`, tid = `2`}, delta: `30`
* (T<sub>1</sub>, T<sub>2</sub>]
* dimensions: {pid = `1001`, tid = `1`}, delta: `3`
* dimensions: {pid = `1001`, tid = `2`}, delta: `8`
* (T<sub>2</sub>, T<sub>3</sub>]
* dimensions: {pid = `1001`, tid = `1`}, delta: `3`
* dimensions: {pid = `1001`, tid = `2`}, delta: `4`
* (T<sub>3</sub>, T<sub>4</sub>]
* dimensions: {pid = `1001`, tid = `1`}, delta: `4`
* dimensions: {pid = `1001`, tid = `2`}, delta: `5`
* (T<sub>4</sub>, T<sub>5</sub>]
* dimensions: {pid = `1001`, tid = `2`}, delta: `6`
* dimensions: {pid = `1001`, tid = `3`}, delta: `5`

You can see that we will have to remember the last value of **every single
permutation we've encountered so far**, because if we don't, we won't be able to
calculate the delta value using `current value - last value`. And you can tell,
this is super expensive.

Making it more interesting, if we have min/max value, it is **mathematically
impossible** to reliably deduce the Delta temporality from Cumulative
temporality. For example:

* If the maximum value is 10 during (T<sub>0</sub>, T<sub>2</sub>] and the
maximum value is 20 during (T<sub>0</sub>, T<sub>3</sub>], we know that the
maximum value during (T<sub>2</sub>, T<sub>3</sub>] must be 20.
* If the maximum value is 20 during (T<sub>0</sub>, T<sub>2</sub>] and the
maximum value is also 20 during (T<sub>0</sub>, T<sub>3</sub>], we wouldn't
know what the maximum value is during (T<sub>2</sub>, T<sub>3</sub>], unless
we know that there is no value (count = 0).
reyang marked this conversation as resolved.
Show resolved Hide resolved

So here are some suggestions that we encourage SDK implementers to consider:

* You probably don't want to encourage your users to do Cumulative to Delta
conversion. Actually, you might want to discourage them from doing this.
* If you have to do Cumulative to Delta conversion, and you encountered min/max,
rather than drop the data on the floor, you might want to convert them to
something useful - e.g. [Gauge](./datamodel.md#gauge).