Always provide the Elasticsearch output unit configuration to Heartbeat when using the shipper

[cmacknz]

Heartbeat has a new feature where it will make queries directly to the Elasticsearch configured as the output. To do this it creates an internal Elasticsearch client from the Elasticsearch output settings provided to the beat. Additionally, Heartbeat only supports the Elasticsearch output and all other outputs types should be removed from the v1 and v2 agent spec files.

When Heartbeat is configured to use the shipper, the agent must provide both the shipper output configuration and the Elasticsearch output configuration.

CC @andrewvc @emilioalvap

[cmacknz]

@fearful-symmetry can you comment on how Heartbeat should handle two provided unit configurations to avoid instantiating a separate Beat Elasticsearch output when the shipper is configured as the output for events?

[fearful-symmetry]

So, we'll need a few changes in the beats V2 output handler to deal with this. The registration system needs to understand that more than one output can be registered, and the v2 API handlers need to be able to send and reload more than one output config.

After that, it should be relatively simple. Heartbeat will call the MustRegisterOutput() method during setup with a function callback for their ES client which satisfies the Reloadable interface:

type Reloadable interface {
	Reload(config *ConfigWithMeta) error
}

The most awkward part of this is is going to be namespacing. Each output config has a "type" key that'll be something like kafka or elasticsearch or whatever. If possible, that should be something unique like heartbeat-es. If elastic-agent can't manage setting a special type key, we'll need a sort of function registration system, similar to what happens for input config transformation, so heartbeat can interrupt the output reloading process and figure out what it should do with a given config based on its own heuristics.

[cmacknz]

I don't think resolving this is urgent, as Heartbeat can simply not support the shipper until we come up with a way to support this on the agent side and it will continue to receive the Elasticsearch output configuration as it does today.

@fearful-symmetry are there still changes necessary to Heartbeat if they only receive a single output unit configuration, and it is always the Elasticsearch output?

[fearful-symmetry]

are there still changes necessary to Heartbeat if they only receive a single output unit configuration, and it is always the Elasticsearch output?

Right now their ES client handler thing is initialized outside of the reloaders, so they won't work under any version of elastic-agent.

Disregarding any integration with the shipper, most of what I mentioned above still applies, as we still need a way for the output config to get routed back to heartbeat.

Off the top of my head, I don't think there's any good pre-existing APIs that would make this particularly easy; if we don't want to go around tinkering with the V1 and V2 reloader system, I wonder if we can somehow expand the elasticsearch client APIs in libbeat so that heartbeat can just import a pre-configured handler somehow.

[cmacknz]

OK thanks, I feel like we could use an issue to track ensuring heartbeat's ES client works at all with v2 configuration system but it sounds like it is effectively covered by elastic/beats#33244.

This issue (ensuring they still have access to the ES output configuration with the shipper) can still just be deferred by having Heartbeat not declare it supports the shipper in the spec file.

I wouldn't try to cheat the configuration reload system, that will probably just cause more problems down the road. When the ES output configuration changes, the ES client likely needs to be recreated just like the ES output itself would be.

[fearful-symmetry]

This issue (ensuring they still have access to the ES output configuration with the shipper) can still just be deferred by having Heartbeat not declare it supports the shipper in the spec file.

Just to be clear, this isn't so much a shipper problem, as much as it is a central management client problem; we currently don't have any way to route the output config to anything other than the pre-defined output. Once we have that, the shipper is sort of a non-issue, assuming elastic-agent doesn't have any problem with sending two output configs.

[jlind23]

The shipper project has been revisited hence closing this as outdated/not relevant.