Allow the Elastic Agent to run the shipper

[cmacknz]

The data shipper must be runnable by the Elastic agent. This issue captures the shipper side work for elastic/elastic-agent#214.

The shipper executable must implement the Elastic agent V2 control protocol, defined here: https://github.com/elastic/elastic-agent-client. Note that the control protocol is being updated as part of the Agent V2 architecture in elastic/elastic-agent-client#29. The control protocol features to be implemented must be coordinated with the control plane team.

While we should use the V2 protocol in the final version of the shipper, initial prototyping can be done using the V1 implementation to unblock progress while the control plane team finalizes the V2 protocol:

• Prototype running the shipper under agent using the V1 protocol #46
• Provide the relevant subset of the agent policy to the shipper as its configuration elastic-agent#617

This issue is considered complete when:

• A test proves that the Elastic agent can successfully start and monitor the data shipper, including logs, metrics, and runtime health.
• A test proves that the Elastic agent can upgrade the data shipper.
• A test proves that the Elastic agent can detect when the shipper is unhealthy or stopped.

[cmacknz]

Assigning this to @fearful-symmetry as it aligns with the metrics/diagnostic reporting work.

[fearful-symmetry]

Okay, brief update:
I don't know if it's tracked on github, but there's a new spec for how agent will bundle + build it's related components, which will probably impact a lot of how this feature will be tested+developed.

The agent-side V2 PR is here: elastic/elastic-agent#502

[cmacknz]

Looks like the V2 spec PR is merged so you should be unblocked.

[fearful-symmetry]

Current state: Have this PR open to make the shipper a tad easier to test: elastic/elastic-agent-client#33

Currently, I have the basic skeleton of a controller client set up and working. Still trying to understand the control flow/startup process.