[processor/metricstransform] Allow scaling exponential histograms

[TylerHelmuth]

Component(s)

metricstransformprocessor

Is your feature request related to a problem? Please describe.

To help normalize telemetry between old and stable http semantic conventions, I need to be able to convert the http.client.duration histogram I need to be able to scale from ms to seconds. The metrics transform processor can do this for histograms, but not exponential histograms.

Describe the solution you'd like

Update the processor so it can scale exponential histograms.

Describe alternatives you've considered

No response

Additional context

open-telemetry/semantic-conventions#534

[github-actions]

Pinging code owners:

• processor/metricstransform: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners for processor/metricstransform: @dmitryax. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

pinging @jmacd @kentquirk @dashpole

[TylerHelmuth]

pinging @jpkrohling

[jpkrohling]

I pinged a few folks here at Grafana Labs who might be interested in the topic.

[aknuds1]

I will look into this when I'm back from PTO, i.e. Tuesday 6th of February.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/metricstransform: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[aknuds1]

Note to self: Look into this.

[jpkrohling]

@sh0rez, this might be something you could help with as well.

[sh0rez]

for clarification, we are talking about scaling the time values, not changing the exponential histograms scale property (the bucket widths), right?

scale and rescale have very specific meaning for exponential histograms, so lets maybe reword this issue a little?

[danelson]

In the older version of the semantic conventions, auto instrumentation produced a metric named http.server.duration which used a unit of ms. In never versions it produces a metric named http.server.request.duration with a unit of s.

Here is a contrived example in .NET.

using System.Diagnostics.Metrics;
using OpenTelemetry;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;

namespace Example
{
    public class Program
    {
        public static void Main(string[] args)
        {
            using var meter = new Meter("TestMeter");

            using var meterProviderBuilder = Sdk.CreateMeterProviderBuilder()
                .ConfigureResource(resource => resource
                    .AddService("example-service"))
                .AddMeter(meter.Name)
                .AddView((instrument) =>
                {
                    if (instrument.GetType().GetGenericTypeDefinition() == typeof(Histogram<>))
                    {
                        return new Base2ExponentialBucketHistogramConfiguration();
                    }

                    return null;
                })
                .AddOtlpExporter()
                .Build();

            // Older application records like this
            var oldConvention = meter.CreateHistogram<double>("http.server.duration", "ms");
            oldConvention.Record(500);

            // Newer application records like this
            var newConvention = meter.CreateHistogram<double>("http.server.request.duration", "s");
            newConvention.Record(0.5);
        }
    }
}

This produces the following result for me

{
    "name": "http.server.duration",
    "unit": "ms",
    "exponentialHistogram":
    {
        "dataPoints":
        [
            {
                "startTimeUnixNano": "1714499479632475900",
                "timeUnixNano": "1714499479644263300",
                "count": "1",
                "sum": 500,
                "scale": 20,
                "positive":
                {
                    "offset": 9401306,
                    "bucketCounts":
                    [
                        "1"
                    ]
                },
                "negative":
                {},
                "min": 500,
                "max": 500
            }
        ],
        "aggregationTemporality": 2
    }
}

vs

{
    "name": "http.server.request.duration",
    "unit": "s",
    "exponentialHistogram":
    {
        "dataPoints":
        [
            {
                "startTimeUnixNano": "1714499479636591800",
                "timeUnixNano": "1714499479644296800",
                "count": "1",
                "sum": 0.5,
                "scale": 20,
                "positive":
                {
                    "offset": -1048577,
                    "bucketCounts":
                    [
                        "1"
                    ]
                },
                "negative":
                {},
                "min": 0.5,
                "max": 0.5
            }
        ],
        "aggregationTemporality": 2
    }
}

If I have a collector deployment that is sent information from different applications emitting data that follow both old and new conventions it makes it difficult to query the data on the backend. I need to be aware that this exists, hope my query language supports a conversion, etc.

It would be great if the collector could convert the old convention payload into the new convention payload.

You are correct that we are not taking about the scale field but rather a complete conversion of the payload, in this case from ms to s.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/metricstransform: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[wildum]

Hey, I can try to tackle this one :) Because the scaling from one bucket to another is constant: base**(i) / base**(i+1) == base**(i+1) / base**(i+2), we don't need to adjust the scale field. I think that we can just update the offset without touching the buckets.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/metricstransform: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.