[Remote Cluster State] Improve stale file deletion logic

[shiv0408]

Is your feature request related to a problem? Please describe

Currently, we are checking and deleting the stale files on every cluster state update. With this logic, we make a list and get calls to our remote repository in all the cluster state updates. If I have a workload, requiring adding dynamic mappings suppose, it will have lots of cluster state updates in every mapping update. This causes lots of list and get calls to repository, having substantial increase in costs for our usecase.

Describe the solution you'd like

If we move away from cleaning the remote files on every cluster state to a configurable period and number of updates after which this will be triggered, it will provide a better customer experience.
We could configure a default period of x hours and number of updates to y. We will clean up the remote files after y number of cluster state updates or x hours, whatever comes first.

Related component

Cluster Manager

Describe alternatives you've considered

If we go ahead with just a period, then if we have are having large number of cluster state updates, the number of stale files will increase significantly and stay there until the period is over.

Similarly, if we just have clean up after n updates, this also may cause stale files to stay in the repository, if updates are infrequent and state is not updated for a long time.

Additional context

No response

[peternied]

[Triage - attendees 1 2 3 4 5 6 7]
@shiv0408 Thanks for creating this issue, we'd welcome a pull request to improve this functionality.

[reta]

The change was reverted due to flaky tests, reopening the issue