Support customization of throughput relevant settings in APM integration

[simitt]

APM Server standalone allows users to customize throughput relevant settings:

• output.elasticsearch.worker, output.elasticsearch.bulk_max_size
• queue.mem.events, queue.mem.flush.min_events

Elastic Agent with Fleet does not expose the memory queue related settings, but they are important to be aligned with the available resources to optimize processing while not running out of memory.
A discussed approach to support this is to make following changes in libbeat:

• ensure that beats properly restart/reload when output related settings change (output.elasticsearch.worker, output.elasticsearch.bulk_max_size)

• introduce a factor F that can be set per beat (not user editable);

  • F=1: lowest throughput, no load balancing.
  • F = (W + R) with W = len(output.*.hosts) * worker. R sets how many batches can be ready for outputs still waiting for ACKs. R=Whas higher memory usage, but can keep the outputs/queues more saturated improving throughput (otherwise it has to wait for the next N events to be acquired). For reducing memory usage set R=0 or R=1. If dealing with bursts of events, F=1 might make sense. Otherwise F=2*W could be a good default.

• translate libbeat internal queue settings from output relevant settings and F:

  • queue.mem.flush.min_events = bulk_max_size
  • queue.mem.events = bulk_max_size * F
  • set a fixed value for queue.mem.flush.timeout

cc @urso

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[urso]

set a fixed value for queue.mem.flush.timeout

I think it depends on the output. For ES/Redis outputs we can introduce a timeout setting with the output.

The Kafka output has it's own buffering and timeout setting. Maybe we just want to stream events through by setting timeout to 0 (we would need to test if that is beneficial or not).

For LS I would love to have an event streaming protocol without need for batches, yet lumberjack uses batches => a flush timeout makes sense here as well.