Define a feature flag for enabling the shipper in the agent policy, defaulting to false.

[ph]

I should be able to turn on or off the possibility to run Elastic Agent with or without the new shipper.

Deliverables:

• Using an option in the Agent policy, allow someone to configure it via Fleet so multiples Agent can be switched remotely.
• Turn on via an environment variable, so you can try it on a single agent.
• Documentation

[ph]

@jlind23 When the plan is fixed we need to synchronize with the Fleet team to make this option available in fleet.

[jlind23]

@ph what about the timing? Should we sneak it into 8.3 as part of the shipper work?

[ph]

I believe so, note depending on the timing we could have it in the fleet UI in 8.4.

[jlind23]

@blakerouse May I assign this to you as you'll be working on it with Alex?

[blakerouse]

@jlind23 I have assigned myself. Thanks!

[nimarezainia]

@fearful-symmetry & @cmacknz I was wondering where this flag would reside. the description says that it should be in the policy but in reality it's an Output object so perhaps it should be part of the output configuration (?)

Also once feature flag is toggled, we should be exposing all the shipper related configurations UI.

@jlind23 @jen-huang : perhaps we need a Fleet UI representation also?

[jlind23]

@nimarezainia yup, that's true.

[blakerouse]

I am working on this now, so it would be great to have a resolution on where we want to put the key. I think in the output configuration is the best place as it doesn't require any UI work to be done (because we want to default and remove this key in the future anyway). I also think it provides flexibility to use the shipper for 1 output but maybe not for the other.

So I was thinking we could do something like:

shipper:
  enabled: true

Super simple and short version is shipper.enabled: true.

[cmacknz]

I agree, putting the flag in the output section makes sense. Instead of just using shipper.enabled: true is it worth creating a features section? Something like:

features:
  shipper: true

That way we have a place to put other feature flag style configuration in the future.

[nimarezainia]

@blakerouse just to close on this; there are some UI elements that are also Shipper specific and would only make sense when this flag is enabled. We decided that we will go ahead with the UI elements (things like memory queue size, bulk size, disk queue, compressions etc) but obviously what ever the user configures in them will not take effect unless this feature flag is set. We will document this.

It avoid us having breaking changes when we GA and remove the need for this feature flag. Of course the Kafka output is also dependent on this.

[blakerouse]

Being that the fact that a shipper is running will be clearly visible in the elastic-agent status output. I think that we should go with the top-level key of:

shipper:
   ...

So that means in the policy it would look like:

outputs:
  default:
    type: elasticsearch
    hosts: https://localhost:9200
    shipper:
       enabled: true
       queue: {}

This makes it clear that you are configuring the shipper which will be a real component and status reported information inside of the Elastic Agent system.

I think we can even go with if the shipper: key is defined that enabled: true is implied.

[cmacknz]

Yes agreed, we had a meeting today and I think keeping a shipper section makes sense for the reasons Blake listed. We may want to add more shipper specific configuration than just an enabled flag, and there is reason to hide the fact that the shipper exists under the hood.

[nimarezainia]

So still true that the UI for output configs will only take affect when this shipper flag is set?

[blakerouse]

Correct. In the Fleet UI if you defined:

shipper: {}

In the output YAML configuration the shipper would be used (only for inputs thats say they support the shipper as well). So let's say starting out that heartbeat doesn't support the shipper, it will still use its internal elasticsearch output and ignore the shipper flags. Then in the next release if it supports it, then it will just start using it.

[jlind23]

@cmacknz @pierrehilbert the PR was merged, shouldn't we close this issue then?

[blakerouse]

Yes, complete!