Migrate filebeat and metricbeat to the V2 agent specification file format

[cmacknz]

As part of the V2 architecture changes the agent specification file format is being redesigned to remove automatic configuration transformations like inject_stream_processor.

Both filebeat and metricbeat currently rely on a variety of configuration file transformations when run under agent which must be removed:

• https://github.com/elastic/elastic-agent/blob/main/internal/spec/filebeat.yml
• https://github.com/elastic/elastic-agent/blob/main/internal/spec/metricbeat.yml

The scope of this issue is to move the logic for each of these configuration transformations into the beat itself when it detects it is being run under agent (for example, seeing the -management.enabled flag on the command line).

Relates To:

• [Meta] Remove Elastic Agent V1 control protocol elastic-agent-client#32
• PR including reference V2 specification files: [v2] Add v2 component specification and validation. elastic-agent#502
• Inject Beat gRPC stream identifiers without relying on agent specification file transformations #33135

This task is considered complete when a test exists proving that filebeat and metricbeat behave correctly when started using their V2 specification file. This includes but is not limited to testing that:

• There were no unexpected changes to the format of published documents.
• Each available input defined in the specification file can be started.
• Policy changes adding processors, inputs, or new output configurations take effect as expected.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

Assigning to @rdner as I think the work here will probably be a natural continuation of #33135 for the shipper.

[cmacknz]

There is a chance this overlaps with the work needed for #31901, ideally they can be done independently but we'll need to sync up first.

[cmacknz]

I think it will make sense for @fearful-symmetry to take this over as it is linked strongly to #31901

[fearful-symmetry]

Yah, still trying to figure out how to grapple with this.

@cmacknz I assume the goal here is to treat the V2 config system as the "default" for how things work, meaning we make the beats backwards-compatible, and future V2 imputs need no config transformation to start with.

I'm undecided on if we can/should "port" that transformation code/spec config into the beats, and have them basically apply the transformations once we get the config from agent. The alternative would be something more piecemeal with custom code for each beat.

[cmacknz]

@cmacknz I assume the goal here is to treat the V2 config system as the "default" for how things work, meaning we make the beats backwards-compatible, and future V2 imputs need no config transformation to start with.

👍 Yes, we want to obsolete the V1 spec files and only use the V2 specs that do not allow transformations.