Many memory allocations with multiple output plugins

[LarsStegman]

Relevant telegraf.conf

[[outputs.influxdb_v2]]
  tagpass = {"batch"="0"}
  ...
[[outputs.influxdb_v2]]
  tagpass = {"batch"="1"}
  ...
[[outputs.influxdb_v2]]
  tagpass = {"batch"="2"}
  ...
...

# Reads and parses about 40kb of data at 50Hz
[[inputs.socket_listener]]
  ...

### Logs from Telegraf

```text
No relevant entries

System info

Telegraf 1.31.2, Ubuntu 24.02

Docker

No response

Steps to reproduce

1. Receive 40KB of data at 50Hz
2. Telegraf will not be able to keep up and will start dropping UDP packets

...

Expected behavior

Telegraf is able to keep up with the amount of data.

Actual behavior

Telegraf drops several of the UDP packets because it is waiting on GOGC.

I profiled Telegraf and noticed that about 28% of all allocations were done by the metric.Copy function called from agent.runOutputs at

telegraf/agent/agent.go

Line 879 in ffee74c

output.AddMetric(metric.Copy())

. This seemed strange to me because each metric should only be output to 1 output plugin. Indeed, when I move the Copy call over to the runningOutput.AddMetric after the metric is filtered out by a filter, the amount of allocations drops drastically and performance increases. (this is a hack and should be implemented in a different way!)

I think it might be worth it to change this mechanism and only copy a metric when an output will accept a metric, i.e. it is not filtered by a Filter.

Additional info

heap
profile

profiles.zip

[srebhan]

Thanks for your report and PR!