statsd receiver does not display well for statsd_type `timing` when observed as observer_type `summary` #29508

charz · 2023-11-27T07:07:34Z

Component(s)

receiver/statsd

What happened?

Description

The statsd receiver does not display well for statsd_type timing when observed as observer_type summary. I observe that <name>_timing_count is consistently 1, and <name>_timing_sum is incorrect.

Steps to Reproduce

The configuration

receivers:
  statsd:
    endpoint: "0.0.0.0:8125" # default
    aggregation_interval: 10s  # default
    enable_metric_type: false  # default
    is_monotonic_counter: true # default
    timer_histogram_mapping:
      - statsd_type: "histogram"
        observer_type: "histogram"
        histogram:
          max_size: 50
      - statsd_type: "timing"
        observer_type: "summary"
      - statsd_type: "distribution"
        observer_type: "histogram"
        histogram:
          max_size: 50


exporters:
  prometheus:
    endpoint: "0.0.0.0:9105"
    send_timestamps: false
    metric_expiration: 120m
    enable_open_metrics: false
    add_metric_suffixes: false
    resource_to_telemetry_conversion:
      enabled: true

processors:
  batch:

service:
  telemetry:
    logs:
      level: debug
    metrics:
      address: ":8889"
  pipelines:
    metrics:
     receivers: [statsd]
     exporters: [prometheus]

The statsd metrics from Ncat

$ date; echo "test.metric.ts:42|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:12|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:1|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; date
Mon Nov 27 06:46:47 UTC 2023
Mon Nov 27 06:46:47 UTC 2023

$ date; echo "test.metric.ts:42|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:12|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:1|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; date
Mon Nov 27 06:47:09 UTC 2023
Mon Nov 27 06:47:09 UTC 2023

Expected Result

Here are the expected results from statsd_exporter. I'm unsure why test_metric_ts_count in the statsd receiver is consistently 1.

The same test agaist statsd_exporter

$ date; echo "test.metric.ts:42|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8133; echo "test.metric.ts:12|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8133; echo "test.metric.ts:1|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8133; date
Mon Nov 27 06:54:58 UTC 2023
Mon Nov 27 06:54:58 UTC 2023

$ curl -sS 127.0.0.1:9102/metrics |grep test.metric.ts; date
# HELP test_metric_ts Metric autogenerated by statsd_exporter.
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0.5"} 0.012
test_metric_ts{myKey="myVal",quantile="0.9"} 0.042
test_metric_ts{myKey="myVal",quantile="0.99"} 0.042
test_metric_ts_sum{myKey="myVal"} 0.05500000000000001
test_metric_ts_count{myKey="myVal"} 3
Mon Nov 27 06:55:00 UTC 2023

$ date; echo "test.metric.ts:42|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8133; echo "test.metric.ts:12|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8133; echo "test.metric.ts:1|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8133; date
Mon Nov 27 06:55:06 UTC 2023
Mon Nov 27 06:55:06 UTC 2023

$ curl -sS 127.0.0.1:9102/metrics |grep test.metric.ts; date
# HELP test_metric_ts Metric autogenerated by statsd_exporter.
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0.5"} 0.012
test_metric_ts{myKey="myVal",quantile="0.9"} 0.042
test_metric_ts{myKey="myVal",quantile="0.99"} 0.042
test_metric_ts_sum{myKey="myVal"} 0.11
test_metric_ts_count{myKey="myVal"} 6
Mon Nov 27 06:55:09 UTC 2023

Actual Result

$ date; echo "test.metric.ts:42|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:12|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:1|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; date
Mon Nov 27 06:46:47 UTC 2023
Mon Nov 27 06:46:47 UTC 2023

$ curl -sS 127.0.0.1:9105/metrics; date
# HELP test_metric_ts
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0"} 1
test_metric_ts{myKey="myVal",quantile="0.1"} 1
test_metric_ts{myKey="myVal",quantile="0.5"} 1
test_metric_ts{myKey="myVal",quantile="0.9"} 1
test_metric_ts{myKey="myVal",quantile="0.95"} 1
test_metric_ts{myKey="myVal",quantile="1"} 1
test_metric_ts_sum{myKey="myVal"} 1
test_metric_ts_count{myKey="myVal"} 1
Mon Nov 27 06:46:59 UTC 2023

$ date; echo "test.metric.ts:42|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:12|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; echo "test.metric.ts:1|ms|@1|#myKey:myVal" | nc -w 1 -u localhost 8125; date
Mon Nov 27 06:47:09 UTC 2023
Mon Nov 27 06:47:09 UTC 2023

$ curl -sS 127.0.0.1:9105/metrics; date
# HELP test_metric_ts
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0"} 1
test_metric_ts{myKey="myVal",quantile="0.1"} 1
test_metric_ts{myKey="myVal",quantile="0.5"} 1
test_metric_ts{myKey="myVal",quantile="0.9"} 1
test_metric_ts{myKey="myVal",quantile="0.95"} 1
test_metric_ts{myKey="myVal",quantile="1"} 1
test_metric_ts_sum{myKey="myVal"} 1
test_metric_ts_count{myKey="myVal"} 1
Mon Nov 27 06:47:17 UTC 2023

$ curl -sS 127.0.0.1:9105/metrics; date
# HELP test_metric_ts
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0"} 1
test_metric_ts{myKey="myVal",quantile="0.1"} 1
test_metric_ts{myKey="myVal",quantile="0.5"} 1
test_metric_ts{myKey="myVal",quantile="0.9"} 1
test_metric_ts{myKey="myVal",quantile="0.95"} 1
test_metric_ts{myKey="myVal",quantile="1"} 1
test_metric_ts_sum{myKey="myVal"} 1
test_metric_ts_count{myKey="myVal"} 1
Mon Nov 27 06:47:25 UTC 2023

$ curl -sS 127.0.0.1:9105/metrics; date
# HELP test_metric_ts
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0"} 1
test_metric_ts{myKey="myVal",quantile="0.1"} 1
test_metric_ts{myKey="myVal",quantile="0.5"} 1
test_metric_ts{myKey="myVal",quantile="0.9"} 1
test_metric_ts{myKey="myVal",quantile="0.95"} 1
test_metric_ts{myKey="myVal",quantile="1"} 1
test_metric_ts_sum{myKey="myVal"} 1
test_metric_ts_count{myKey="myVal"} 1
Mon Nov 27 06:47:36 UTC 2023

Collector version

v0.88.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  statsd:
    endpoint: "0.0.0.0:8125" # default
    aggregation_interval: 10s  # default
    enable_metric_type: false  # default
    is_monotonic_counter: true # default
    timer_histogram_mapping:
      - statsd_type: "histogram"
        observer_type: "histogram"
        histogram:
          max_size: 50
      - statsd_type: "timing"
        observer_type: "summary"
      - statsd_type: "distribution"
        observer_type: "histogram"
        histogram:
          max_size: 50


exporters:
  prometheus:
    endpoint: "0.0.0.0:9105"
    send_timestamps: false
    metric_expiration: 120m
    enable_open_metrics: false
    add_metric_suffixes: false
    resource_to_telemetry_conversion:
      enabled: true

processors:
  batch:

service:
  telemetry:
    logs:
      level: debug
    metrics:
      address: ":8889"
  pipelines:
    metrics:
     receivers: [statsd]
     exporters: [prometheus]

Log output

$ ./otelcol-contrib --config otel.yaml; date
2023-11-27T06:46:44.441Z	info	service@v0.88.0/telemetry.go:84	Setting up own telemetry...
2023-11-27T06:46:44.441Z	info	service@v0.88.0/telemetry.go:201	Serving Prometheus metrics	{"address": ":8889", "level": "Basic"}
2023-11-27T06:46:44.441Z	debug	exporter@v0.88.0/exporter.go:273	Beta component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:44.441Z	debug	receiver@v0.88.0/receiver.go:294	Beta component. May change in the future.	{"kind": "receiver", "name": "statsd", "data_type": "metrics"}
2023-11-27T06:46:44.446Z	info	service@v0.88.0/service.go:143	Starting otelcol-contrib...	{"Version": "0.88.0", "NumCPU": 48}
2023-11-27T06:46:44.446Z	info	extensions/extensions.go:33	Starting extensions...
2023-11-27T06:46:44.446Z	warn	internal@v0.88.0/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "exporter", "data_type": "metrics", "name": "prometheus", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-11-27T06:46:44.447Z	info	service@v0.88.0/service.go:169	Everything is ready. Begin running and processing data.
2023-11-27T06:46:50.172Z	debug	prometheusexporter@v0.88.0/collector.go:362	collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:50.172Z	debug	prometheusexporter@v0.88.0/accumulator.go:268	Accumulator collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:54.041Z	debug	prometheusexporter@v0.88.0/collector.go:362	collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:54.041Z	debug	prometheusexporter@v0.88.0/accumulator.go:268	Accumulator collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:54.448Z	debug	prometheusexporter@v0.88.0/accumulator.go:79	accumulating metric: test.metric.ts	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:54.448Z	debug	prometheusexporter@v0.88.0/accumulator.go:79	accumulating metric: test.metric.ts	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:54.448Z	debug	prometheusexporter@v0.88.0/accumulator.go:79	accumulating metric: test.metric.ts	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:59.666Z	debug	prometheusexporter@v0.88.0/collector.go:362	collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:59.666Z	debug	prometheusexporter@v0.88.0/accumulator.go:268	Accumulator collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:46:59.666Z	debug	prometheusexporter@v0.88.0/collector.go:386	metric served: Desc{fqName: "test_metric_ts", help: "", constLabels: {}, variableLabels: {myKey}}	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:14.448Z	debug	prometheusexporter@v0.88.0/accumulator.go:79	accumulating metric: test.metric.ts	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:14.448Z	debug	prometheusexporter@v0.88.0/accumulator.go:79	accumulating metric: test.metric.ts	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:14.448Z	debug	prometheusexporter@v0.88.0/accumulator.go:79	accumulating metric: test.metric.ts	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:17.351Z	debug	prometheusexporter@v0.88.0/collector.go:362	collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:17.351Z	debug	prometheusexporter@v0.88.0/accumulator.go:268	Accumulator collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:17.351Z	debug	prometheusexporter@v0.88.0/collector.go:386	metric served: Desc{fqName: "test_metric_ts", help: "", constLabels: {}, variableLabels: {myKey}}	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}

2023-11-27T06:47:25.566Z	debug	prometheusexporter@v0.88.0/collector.go:362	collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:25.566Z	debug	prometheusexporter@v0.88.0/accumulator.go:268	Accumulator collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:25.566Z	debug	prometheusexporter@v0.88.0/collector.go:386	metric served: Desc{fqName: "test_metric_ts", help: "", constLabels: {}, variableLabels: {myKey}}	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:36.208Z	debug	prometheusexporter@v0.88.0/collector.go:362	collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:36.208Z	debug	prometheusexporter@v0.88.0/accumulator.go:268	Accumulator collect called	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-11-27T06:47:36.208Z	debug	prometheusexporter@v0.88.0/collector.go:386	metric served: Desc{fqName: "test_metric_ts", help: "", constLabels: {}, variableLabels: {myKey}}	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}

^C2023-11-27T06:47:41.510Z	info	otelcol@v0.88.0/collector.go:250	Received signal from OS	{"signal": "interrupt"}
2023-11-27T06:47:41.510Z	info	service@v0.88.0/service.go:178	Starting shutdown...
2023-11-27T06:47:41.510Z	debug	statsdreceiver@v0.88.0/reporter.go:97	UDP Transport ([::]:8125) - ReadFrom error: read udp [::]:8125: use of closed network connection	{"kind": "receiver", "name": "statsd", "data_type": "metrics"}
2023-11-27T06:47:41.510Z	info	extensions/extensions.go:50	Stopping extensions...
2023-11-27T06:47:41.510Z	info	service@v0.88.0/service.go:192	Shutdown complete.
Mon Nov 27 06:47:41 UTC 2023

Additional context

No response

The text was updated successfully, but these errors were encountered:

github-actions · 2023-11-27T07:07:51Z

Pinging code owners:

receiver/statsd: @jmacd @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

atoulme · 2023-11-29T17:49:26Z

@charz thank you for this report, would you like to try to fix this issue?

charz · 2023-12-05T10:52:52Z

@atoulme Do you have any guidelines that I should follow?

atoulme · 2023-12-05T16:43:49Z

Sure, I'd take your tests and curl requests and turn this into a test that fails. From there, I'd open a PR to show the failing test and ask for help to resolve, or try to implement a fix.

charz · 2023-12-27T16:25:56Z

Just figured it out. Will send a PR later.

➜  ~ echo "test.metric.ts:22|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:22|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:22|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:22|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:2|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:2|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:2|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:2|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:2|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:2|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;
➜  ~ echo "test.metric.ts:2|ms|@1|#myKey:myVal" | nc -w 1 -u 127.0.0.1 8125;

➜  ~ curl -sS 127.0.0.1:9105/metrics
# HELP test_metric_ts
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0"} 2
test_metric_ts{myKey="myVal",quantile="0.1"} 2
test_metric_ts{myKey="myVal",quantile="0.5"} 22
test_metric_ts{myKey="myVal",quantile="0.9"} 22
test_metric_ts{myKey="myVal",quantile="0.95"} 22
test_metric_ts{myKey="myVal",quantile="1"} 22
test_metric_ts_sum{myKey="myVal"} 140
test_metric_ts_count{myKey="myVal"} 10

➜  ~ curl -sS 127.0.0.1:9105/metrics
# HELP test_metric_ts
# TYPE test_metric_ts summary
test_metric_ts{myKey="myVal",quantile="0"} 2
test_metric_ts{myKey="myVal",quantile="0.1"} 2
test_metric_ts{myKey="myVal",quantile="0.5"} 2
test_metric_ts{myKey="myVal",quantile="0.9"} 2
test_metric_ts{myKey="myVal",quantile="0.95"} 2
test_metric_ts{myKey="myVal",quantile="1"} 2
test_metric_ts_sum{myKey="myVal"} 6
test_metric_ts_count{myKey="myVal"} 3

The receiver uses each metric's address as the key to create instruments, but the port in the address of each metric is dynamically assigned. This results in the StatsD receiver creating numerous instruments and being unable to aggregate metrics together. Fixes open-telemetry#29508, fixex open-telemetry#23809

github-actions · 2024-02-26T03:29:26Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

receiver/statsd: @jmacd @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions · 2024-04-26T05:18:56Z

This issue has been closed as inactive because it has been stale for 120 days with no activity.

charz added bug Something isn't working needs triage New item requiring triage labels Nov 27, 2023

github-actions bot added the receiver/statsd statsd related issues label Nov 27, 2023

github-actions bot mentioned this issue Nov 28, 2023

Weekly Report: 2023-11-21 - 2023-11-28 #29517

Closed

98 tasks

atoulme removed the needs triage New item requiring triage label Nov 29, 2023

This was referenced Dec 28, 2023

Fix statsd receiver aggregate metrics in various instruments. #30228

Closed

[statsdreceiver] incorrect aggregation when several producers report the same metrics #23809

Closed

github-actions bot added the Stale label Feb 26, 2024

github-actions bot added the closed as inactive label Apr 26, 2024

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statsd receiver does not display well for statsd_type `timing` when observed as observer_type `summary` #29508

statsd receiver does not display well for statsd_type `timing` when observed as observer_type `summary` #29508

charz commented Nov 27, 2023

github-actions bot commented Nov 27, 2023

atoulme commented Nov 29, 2023

charz commented Dec 5, 2023

atoulme commented Dec 5, 2023

charz commented Dec 27, 2023

github-actions bot commented Feb 26, 2024

github-actions bot commented Apr 26, 2024

statsd receiver does not display well for statsd_type timing when observed as observer_type summary #29508

statsd receiver does not display well for statsd_type timing when observed as observer_type summary #29508

Comments

charz commented Nov 27, 2023

Component(s)

What happened?

Description

Steps to Reproduce

The configuration

Expected Result

The same test agaist statsd_exporter

Actual Result

Collector version

Environment information

Environment

OpenTelemetry Collector configuration

Log output

Additional context

github-actions bot commented Nov 27, 2023

atoulme commented Nov 29, 2023

charz commented Dec 5, 2023

atoulme commented Dec 5, 2023

charz commented Dec 27, 2023

github-actions bot commented Feb 26, 2024

github-actions bot commented Apr 26, 2024

statsd receiver does not display well for statsd_type `timing` when observed as observer_type `summary` #29508

statsd receiver does not display well for statsd_type `timing` when observed as observer_type `summary` #29508