Beats run under agent should start without reading a configuration file from disk.

[cmacknz]

For compatibility with the V2 agent control protocol, beats run under agent should start without loading a configuration file from disk. Preferably a beat run under agent should start, establish a V2 control protocol client, and wait for an initial configuration from the agent.

This issue is cloned from @michalpristas's original comment in the V1 protocol removal issue: elastic/elastic-agent-client#32 (comment)

Beats behaviour change

Current

At the moment we handle process as this:

• elastic agent starts
• elastic agent is notified about or detects config change
• elastic agent creates a specific change for specific beats (let's assume filebeat from now on)
• elastic agent starts filebeat process with -management.enabled flag
• filebeat process starts
• filebeat loads a config from filebeat.yml

	cfg, err := cfgfile.Load("", settings.ConfigOverrides)
	if err != nil {
		return fmt.Errorf("error loading config file: %w", err)
	}

• filebeat initializes manager to get comms with agent going

	// initialize config manager
	b.Manager, err = management.Factory(b.Config.Management)(b.Config.Management, reload.Register, b.Beat.Info.ID)
	if err != nil {
		return err
	}

• filebeat fetches config from fleet

Proposed

• elastic agent starts
• elastic agent is notified about or detects config change
• elastic agent creates a specific change for specific beats (let's assume filebeat from now on)
• elastic agent starts filebeat process with -management.enabled flag
• filebeat process starts
• in case management.enabled flag is not set filebeat loads a config from filebeat.yml otherwise use sane default configuration

	cfg, err := cfgfile.Load("", settings.ConfigOverrides)
	if err != nil {
		return fmt.Errorf("error loading config file: %w", err)
	}

• filebeat initializes manager to get comms with agent going

	// initialize config manager
	b.Manager, err = management.Factory(b.Config.Management)(b.Config.Management, reload.Register, b.Beat.Info.ID)
	if err != nil {
		return err
	}

• filebeat fetches config from fleet2

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

Switching the assigning to @fearful-symmetry here as they are likely free next, and this will line up more closely with the input V2 work than the shipper work.

[cmacknz]

There is a chance this overlaps with the work needed for #31900, ideally they can be done independently but we'll need to sync up first.

[simitt]

Is this going to be handled purely in libbeat or do users of libbeat need to adapt their logic?

APM Server is currently making use of cfgfile.ConditionalOverride in two places (e.g. https://github.com/elastic/apm-server/blob/main/internal/beatcmd/root.go#L42-L68) - will this still be supported (also for fleet managed mode)?

cc @axw

[cmacknz]

It should be handled in libbeat, see the change to libbeat/cfgfile/cfgfile.go in #32673.

Essentially instead of reading the configuration from disk it just constructs a default configuration in place with config.NewConfig() before applying any overrides. I'll make a note on the PR to confirm, but I believe the overrides should still apply