feat(processors.batch): Add batch processor

[LarsStegman]

Summary

This new processor can distribute metrics across batches by adding a tag indicating what batch number it is in. This makes it possible to distribute the load of a high number of metrics across multiple instances of the same output plugin.

Checklist

• No AI generated code was used in this PR

Related issues

resolves #15621
resolves #11707

[LarsStegman]

@srebhan the implementation is a little different than what you suggested. I did not see the added benefit of also specifying the batch size, since any overflow would probably just overflow into the next batch, which then also overflows. Please let me know if it should still be added.

[srebhan]

Thanks @LarsStegman for the contribution! Just two small comments from my side. Furthermore, should we also add a force_rebatch option that will only overwrite the batch tag if it does not already exists? I'm asking because in the current default, Telegraf will run each processor twice, once before and once after aggregators if any.

[LarsStegman]

should we also add a force_rebatch option that will only overwrite the batch tag if it does not already exists? I'm asking because in the current default, Telegraf will run each processor twice, once before and once after aggregators if any.

Hmmm interesting. The results after the second pass will indeed be different, because the processor will already have run a pass and the count will have increased. I think it is better to add that feature indeed. It will be more predictable for users.

[LarsStegman]

I made the rebatching enabled by default, because it is less computational load. By default it will now not check the existing tags.

[srebhan]

Two more comments. Regarding the flag, I'm fine either way but slightly tend to your approach...

[srebhan]

@LarsStegman awesome! Maybe just avoid abbreviations in config options? How about naming this just batches?

[LarsStegman]

@srebhan looks like the test runner timed out or something

[srebhan]

Thanks @LarsStegman!

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

🥳 This pull request decreases the Telegraf binary size by -8.01 % for linux amd64 (new size: 239.8 MB, nightly size 260.6 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

[knollet]

But, to parallelize processing, this doesn't help to create batch instances of, say, some regex processor, right? It only does the round-robing tagging.

Also: Does this help as efficiently as possible? Because with this, all the metrics still go down the same pipeline (and by this: go-channel) and don't split up into multiple batch processing pipelines. Every metric still has to be sorted out by a metric-/tagpass deciding "no this metric isn't batch-tagged for me" on every processor defined.

I have to say: I don't see this resolving any one of the claimed Feature-Requests.
This just casts a really bad duct-tape-solution made from starlark into an equally bad duct-tape-solution made from golang.

There shouldn't be batching-tags but

1. the slow processors should be made parallely spawnable.
2. The distribution into these parallely spawned processor instances should be realized by multiple go-channel-consumers.

[LarsStegman]

@knollet I agree, this processor does not fix #11707. @srebhan can that issue be reopened? I see you added it to this PR.

[knollet]

I mean, If I batch into, lets say, 2, I have to duplicate my complex code...

[[processors.batch]]
  batch_tag="batch"
  batches = 2
  
[[processors.regex]]
  tagpass = { "batch" = [ "0" ] }  # "0", not 0... srsly?
  ...some slow regex stuff...
  
# I have to replicate this here... exactly, no mistakes allowed, batches - 1 times, 
# else there's not even a resemblence of parallel processing. hard to catch bugs, 
# even if you generate this with a templating engine like jinja.
[[processors.regex]]
  tagpass = { "batch" = [ "1" ] }  # "1", not 1... srsly?
  ...some slow regex stuff...

[LarsStegman]

@knollet this processor was not meant for increasing the efficiency of the processor pipeline, but to increase output capacity of the end of the pipeline. See #15621 (comment) and down.

[knollet]

Yeah, ok. I don't wanna trash talk your contribution.
Still: Don't you have the problem of having to duplicate that which your batch processor feeds into? May it be a processor or an output plugin doesn't really matter, does it?

[srebhan]

@knollet I reopened #11707.