Make snapshot deletes less memory intensive by reordering repository metadata updates

[original-brownbear]

Current snapshot deletes consume memory that scales as O(N) in the number of shards in the repository that the delete operation touches. This makes them very memory intensive when deleting snapshots that themselves contain many shards.
The deeper issue behind this O(N) memory consumption is that snapshot deletes currently update all shard level metadata in the repository before updating root level metadata and then finally running the blob delete operations.
This means that master must hold information for all shards in memory from the beginning of the delete operation until the physical blob delete execution. Moreover, this also means that delete operations block other repository operations for longer, the larger the number of shards in the delete operation because no other operation may execute while the shard level metadata is updated.

We discussed this issue and settled on a fix that will update the root level metadata first and then update the shard level metadata (that might now point to non-existent snapshots) step-by-step in a newly added delete/GC step. This allows updating only a limited number of shards at once, limiting the memory consumption of the operation and shortening the window during which other operations are blocked by delete operations.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)