Prioritizing shard recoveries

[avleen]

Feature request:

When multiple shard replicas need recovery (eg, when you restart the whole cluster), it can be beneficial to get some indices green before others.
In the Logstash case, getting the most recent indices by date.
In other cases, there may be application demands on which index you really need back first.

Being able to set a specific priority on indices, or have it automatically determined for date-based indices, would be quite helpful.

[s1monw]

once we fixed #10032 this will be much easier since we can recover unsealed shards first or even last since all the sealed once are instant and you problem will be going away entirely.

[kadaan]

#10032 may not address the full request as certain indices may be more important for a business then others. Ideally, ES could take this priority into consideration when determining which shards to recover given limited recovery resources.

[clintongormley]

Closing in favour of #11787