[Fleet] Add global component template to all fleet index templates

[nchaulet]

Description

Resolve #101309

With that PR event.ingested should be correctly mapped for all Fleet managed index template.

This PR introduce a new global component template, that is added to all the index template installed by Fleet.
This component template set the final_pipeline to be a global fleet final pipeline that verify agent id against the used API key, and add the mappings for the fields added by the fleet final pipeline.

Assets naming, I choose to name the two assets like this (open to better naming, and to discuss the convention here)

• .fleet_component_template-{version} for the component template
• .fleet_final_pipeline-{version} for the pipeline that check agent status

The feature is expose behind a config flag xpack.fleet.agentIdVerificationEnabled by default set to true.

How it works

The pipeline and the component template name contains a version so we can upgrade them in the future.

During the fleet setup, if the component template or the pipeline is not here we are going to:

• create the pipeline and the component template
• Apply the component template to all the installed integrations (for this I force reinstall the package, it's probably too much work but it used a well known code path and I think it avoid bug risk)
  • this is going to update the index template and update write indices

Future upgrade (not implemented) can

• create new pipeline and new component template
• Apply this to existing integrations
• Delete unused old pipeline and component template

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[ruflin]

Could we put this installation behind a config option? It should be turned on by default but I think we should offer users to turn it off. This also helps in the case we discover any issues with the installation mechanism, there is an opt out.

On the upgrade path, can you make sure we follow up on this? I have the suspicion we need it sooner then we like.

There is a downside to having the pipeline versioned. If it is not versioned, we could just overwrite it all the processing would use the new one. If we version it, we need to update all data streams. Maybe if the pipeline keeps the data the same, we just keep the version? But how will then the upgrade mechanism know to overwrite?

[nchaulet]

Could we put this installation behind a config option? It should be turned on by default but I think we should offer users to turn it off. This also helps in the case we discover any issues with the installation mechanism, there is an opt out.

Just added the xpack.fleet.agentIdVerificationEnabled flag

For the migration

If we want to only update the pipeline we can use the version property to know if we should upgrade it or not, we will have 3 version to track the version in the pipeline name, the version in the pipeline, and the version of the component template, it start to add some complexity.

I am wondering this will be a good idea. If we have a migration system for Fleet close to the one we use for Saved object, with a version saved in one of our saved object, and based on that we can run migration function where we can do whatever we want, it probably can help for other things like endpoint migration between version too. (but there is a risk to have the ES assets out of sync with what we have if a user manually change them)

[ruflin]

Thanks for adding the config. Will this always install the assets but not just enable them? What if a user had it enabled and hits issues and wants to disable it, how does this work?

For the upgrade: Happy to discuss it in a separate thread.

[nchaulet]

Disabling the flag will not install the assets and not enable.

What if a user had it enabled and want to disable

new package installation will not have it but old packages will have to be reinstalled to disable the component template.

[ruflin]

I'm a bit worried about the part that after disabling it is not really disabled. Is there an easy way for users to "reinstall" a package?

[nchaulet]

Is there an easy way for users to "reinstall" a package?

Not really an easy way it's possible to reinstall the package via calling the API to install the package with force.

If you think it's problematic I can do it in the setup call if we detect the flag is off and there is a component template we can clean everything.

[ruflin]

@nchaulet @jen-huang I suggest for now we get this PR in quickly so we have enough time to test and see if it causes any issue. If it does not have any issues we don't need to worry about the disable part. There is also a "hacky" way to disable it which is overwrite the pipeline with an empty pipeline in case of "emergency" ;-)

[nchaulet]

@elasticmachine merge upstream

[jfsiii]

Code LGTM. Left two grammar-related suggestions. I'm w/ @ruflin re: merging and confirming behavior

[nchaulet]

@elasticmachine merge upstream

[nchaulet]

@elastic/ml-ui I add to modify an API test as it's required to setup fleet before installing a package.

[nchaulet]

@elasticmachine merge upstream

[jfsiii]

@nchaulet I just merged #101769 which means this PR needs the update you mentioned to prevent the global template from being removed.

Hopefully the change is clear and simple enough, but please let me know if you have any questions or if I can help at all.

[pheyos]

ML test changes LGTM

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 0108968
• Storybooks Preview
• Documentation Changes

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
fleet	1014	1015	+1

Unknown metric groups

API count

id	before	after	diff
fleet	1106	1107	+1

History

• 💔 Build #133549 failed 1bcc619
• 💔 Build #133529 failed 27b292a
• 💔 Build #133507 failed f7c5b6fe5558c99b10a5e40fd99fd688c24fcd5b
• 💚 Build #133364 succeeded c873b1c
• 💚 Build #133319 succeeded e7c0e23

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @nchaulet

[kibanamachine]

💔 Backport failed

Status	Branch	Result
❌	7.x	Commit could not be cherrypicked due to conflicts

To backport manually run:
node scripts/backport --pr 102225