[Elastic Agent] Support memory queue configuration

[simitt]

The queue.* parameters in combination with output.elasticsearch.bulk_max_size and output.elasticsearch.worker are the most important settings to tune when scaling an APM Server deployment. Therefore it is important that the queue.* params are still supported; it is a requirement for running APM Server managed by Elastic Agent on cloud.

Different queue configurations should be allowed per sub process, as they will expect different workloads. Generally it is recommended to choose the queue settings in alignment with the above mentioned elasticsearch settings. While only one global output can be configured at the moment, the queue settings should still be configurable per sub process.

This probably requires changes in the memqueue to make it reloadable.

cc @ph, @ruflin, @urso

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[urso]

Hm, I would rather see the memory queue to be removed in the future. In context of agent the memory queue would become an output setting, although it is somewhat unfortunated that each Beat will have it's own output.

[simitt]

We will need some support to configure the queue for 7.x when APM Server is managed by Elastic Agent. This is important to be able to scale the APM Server.

[ruflin]

@simitt As a temporary solution, would it be enough to configure it through the input part and apm-server translates it to the correct place?

[simitt]

Configuration wise it would be fine to configure within the APM Server input vars; but we still need to have a solution in beats for actually applying the queue settings to the memory queue. This cannot be done by the APM Server code itself. Probably something similar to libbeat/cmd/instance/beat.go would be required in beats.

[ph]

It is just me or having the queue defined in the input or adding it to the output requires the same amount of work, which to makes the queue reloadable? I don't think restarting the beats is an option here?

[simitt]

I believe the difference would be that currently only one global output can be configured, but the queue size should be configurable per sub process (e.g. having a different queue size for APM Server than for metricbeat, as the event size probably differs etc.).
Agree that applying the queue settings needs to be done in libbeat in both cases.

[ruflin]

Reloading would definitively not be nice but I think today this is what happens with apm-server legacy if the queue config is changed?

[simitt]

Closing this in favor of #26638