[Fleet] Rollover datastream when index template settings changed

[juliaElastic]

Currently Fleet rolls over datastreams if:

• there is an error on mapping update [Ingest Manager] Trigger rollover when mappings conflict during a package update #68381
• tsdb setting changed [Fleet] Rollover data streams when package w/ TSDB setting changed is installed #157869

There are other use cases where rollover would be required on index template settings changes, like non-dynamic changes.

The reason we don’t apply settings changes is because users might have customized settings via a @custom component template.

cc @felixbarny

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[felixbarny]

@dakrone This issue is one reason why an API that applies a given index template or component template to all data streams it applies to would be useful. Depending on whether or not the changes can be applied dynamically, Elasticsearch might perform a rollover on the data stream.

Currently, Fleet tries to do that but I think that's something that should be handled by Elasticsearch. Not only could avoid complexity in Fleet, the way Fleet does it is fundamentally limited in the way it computes the "diff" between the current state of a data stream from a index template that may consist of several component templates.

[joshdover]

Big +1 to having Elasticsearch handle this. Fleet requires too much knowledge about Elasticsearch in order to do this without missing edge cases, in ways that can easily change without us knowing as new features are added to Elasticsearch.

[dakrone]

It sounds similar to elastic/elasticsearch#89346, where the stream is flagged as some kind of change was made that would mean a rollover when the next doc is indexed. But in this case, it sounds like you want ES to automatically determine if the rollover is necessary?

[felixbarny]

While related, there's a bigger underlying issue behind this. I think we need a way to declare the desired state of data stream in an Infrastructure as Code-like way. I've created an issue with more thoughts in the Elasticsearch repo:

• Apply changes to index/component templates to affected data streams elasticsearch#96521