Flag data stream as "needs a new index" to handle mappings updates where no documents are indexed

[dakrone]

Description

Currently when fleet makes a mapping change (for example, a package being updated) that is linked to a set of data streams, it tries to apply the mapping update to the data stream's backing indices. If the mapping update can be made, then everything is great. If the mapping change cannot be made (a non-additive change) then the mappings are updated in the template and a new rollover is forced for the data stream, to make the mappings take effect on a newly created index immediately.

This, however, can lead to many empty indices, because the data stream linked to an integration may not actually be receiving any data. If the package were upgraded 3-4 times, it could result in 3-4 empty indices for the data stream. Since the data stream is manually rolled over (an unconditional rollover request), the work for #86203 won't help, because we will not know that the data stream is not receiving any data.

One idea for fixing this would be to introduce a flag for the data stream that indicated "roll over the data stream the very next time you receive a document." This would allow fleet to update the mappings as many times as necessary without creating any new empty indices. In the event that the data stream never receives a document, then there will still be no empty indices. If the data stream is being used, then a manual rollover will still not be necessary, as the next document after updating this flag would cause the data stream to roll over.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[gmarouli]

Fixed by #103309