[Upgrade assistant] Make the reindex server side logic more robust to failures

[sebelga]

While reviewing #123727 I realised that when we shut Kibana in a middle of a re-index operation and then we start Kibana again, the state of the reindex is corrupted.

Once we are back in UA, it first indicate during 30 seconds "Reindexing in progress 10%" until it finally goes to the "Reindex paused" state.

I uploaded here a gif of the issue (the gif is too big for Github). Notice in the Chrome network tab the loop of all the requests made to the server.

I think it would be good that we revisit the server side code of the Upgrade assistant reindexing and make it more robust to different failures.

Note: to be investigated but while working on #123638 we realised with @sabarasaba that the reindex operation saved objects were not cleaned up after a successful reindex. We probably want to delete those SO after reindexing.

[elasticmachine]

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

[cjcenizal]

I think it would be good that we revisit the server side code of the Upgrade assistant reindexing and make it more robust to different failures.

Just want to note that Upgrade Assistant might look very different after we've redesigned it to support minor upgrades. We could even do something crazy like request specific ES migration APIs so we won't need any reindexing logic at all.