[ML Data Frame] Refactor stop logic (#42644)

[benwtrent]

In AsyncTwoPhaseIndexer. finishAndSetState the onStop and onAbort methods could be called from inside an atomic update. Both those methods are abstract designed to be overridden and that overriding, by an implementor who is not aware of the restrictions (i.e. me), may introduce side effects which are not safe.

I'm not sure what the behaviour is if another thread tries to get() the atomic reference during a call to updateAndGet() but the docs warn against it. I believe this is the cause of the CI failures seen in #42344.

The function should be side-effect-free, since it may be re-applied when attempted updates fail due to contention among threads.

https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/atomic/AtomicReference.html#updateAndGet(java.util.function.UnaryOperator)

The methods onStop and onFinish must be called before state is saved (1 to set the correct Data Frame state and 2 to increment the checkpoint) but this ordering means there is a race between onStop completing the persistent task and doSaveState updating the persistent task parameters. Luckily after #41942 it is no longer necessary to update the persistent task as all state is persisted and restored from the index so I have remove the p. task update from doSaveState.

ClientDataFrameIndexer.onStop was also persisting state which is not necessary as doSaveState is called after onStop by the base class.

There is also a refactoring change to AsyncTwoPhaseIndexer.stop to make it work the same as abort. abort does not call onAbort if the indexer is not running it is left to the client code to handle that, stop should use the same pattern as it is confusing to use 2 different paradigms in the class interface.

Finally this un-mutes tests muted for #42641

Closes #42344

[elasticmachine]

Pinging @elastic/ml-core