CloudWatch metric sink is not sending http_client_rtt_seconds (aggregated histogram) metrics to CloudWatch

[mkwan-amzn]

Hi Everyone,

I'm running Vector 0.37.1 with a configuration that has:

• An internal metrics source
• A HTTP sink
• A CloudWatch sink
• A filter transform step for the internal metrics source to filter out the metrics I want
• A Console sink to stdout

I'm seeing an issue in which a metric that I can see being emitted for my HTTP Sink, http_client_rtt_seconds, is not being emitted to CloudWatch and there are no logs (debug, trace or error) indicating that there was an issue. From STDOUT, I can see that the metric looks like this:

{
    "name": "http_client_rtt_seconds",
    "namespace": "redacted",
    "tags":
    {
        "component_id": "redacted",
        "component_kind": "sink",
        "component_type": "http",
        "status": "200"
    },
    "timestamp": "2024-11-20T18:32:45.508483798Z",
    "kind": "absolute",
    "aggregated_histogram":
    {
        "buckets":
        [
            ... (removed some for spacing)
            {
                "upper_limit": 0.25,
                "count": 0
            },
            {
                "upper_limit": 0.5,
                "count": 1
            },
            {
                "upper_limit": 1.0,
                "count": 0
            },
            ... (removed some for spacing)
        ],
        "count": 1,
        "sum": 0.326666255
    }
}

I looked through the code for the CloudWatch Metrics Sink and saw that these http_client_rtt_seconds metric are defined as a having a MetricValue type of AggregatedHistogram, however the Sink doesn't seem to support that MetricValue type: code ref

I have two questions around this:

1. Is there any way for me to get around the current lack of support for AggregatedHistogram metrics for this sink in the short term?

2. For the long term, does it make sense to add support for AggregatedHistogram metric value types?

Thanks in advance for the help!

[jszwedko]

Indeed, it looks like the AWS CloudWatch sink does not support aggregated histograms:

vector/src/sinks/aws_cloudwatch_metrics/mod.rs

Lines 271 to 308 in 3d7f06d

	match event.value() {
	MetricValue::Counter { value } => Some(
	MetricDatum::builder()
	.metric_name(metric_name)
	.value(*value)
	.set_timestamp(timestamp)
	.set_dimensions(dimensions)
	.build(),
	),
	MetricValue::Distribution {
	samples,
	statistic: _,
	} => Some(
	MetricDatum::builder()
	.metric_name(metric_name)
	.set_values(Some(samples.iter().map(|s| s.value).collect()))
	.set_counts(Some(samples.iter().map(|s| s.rate as f64).collect()))
	.set_timestamp(timestamp)
	.set_dimensions(dimensions)
	.build(),
	),
	MetricValue::Set { values } => Some(
	MetricDatum::builder()
	.metric_name(metric_name)
	.value(values.len() as f64)
	.set_timestamp(timestamp)
	.set_dimensions(dimensions)
	.build(),
	),
	MetricValue::Gauge { value } => Some(
	MetricDatum::builder()
	.metric_name(metric_name)
	.value(*value)
	.set_timestamp(timestamp)
	.set_dimensions(dimensions)
	.build(),
	),
	_ => None,

It should probably log a warning or something when dropping those metrics. I agree it should support them; probably by converting each bucket into a separate counter. Would you want to open a feature request?

[mkwan-amzn]

Thanks for the reply!

Yes makes sense, I can open a feature request when I have a moment. I should do so using this template right? https://github.com/vectordotdev/vector/issues/new?assignees=&labels=type%3A+feature&projects=&template=feature.yml

[jszwedko]

That's right 🎉