Increase timeouts in TimeSeriesLifecycleActionsIT#testWaitForSnapshot and testWaitForSnapshotSlmExecutedBefore test

[probakowski]

This change adds some randomness and cleanup step to TimeSeriesLifecycleActionsIT#testWaitForSnapshot and testWaitForSnapshotSlmExecutedBefore tests in attempt to make them stable.

Reletes to #50781

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[probakowski]

@elasticmachine run elasticsearch-ci/2

[original-brownbear]

@probakowski can you explain what makes this test fail maybe? It's hard for me to gauge this PR without an understanding of the failure.

[dakrone]

@probakowski I can't tell either how this fixes the test, is there a problem with the SLM policy not using the same ID or repository between tests?

[probakowski]

As I can't reproduce it locally (neither can @martijnvg or PR CI) this is in guess territory.
My idea is that maybe hammering the same repo multiple times in the row can lengthen the time it takes to create a snapshot or that taking snapshot in intake job just takes longer.
I've seen some longer timeout (2 minutes) in testDeleteDuringSnapshot so I assume it can be connected. I've extended timeout to be the same in my tests.
All changes from this PR should be safe in the sense that they shouldn't make things worse but they can improve stability.

[original-brownbear]

@probakowski how much of the default timeout that you want to up to 2min here are we using on a normal test run, did you try that out? Maybe there is some cron issue here (just guessing because we had that) and we can reproduce this by moving to half the current timeout or so?

[probakowski]

@original-brownbear default is 10s, in my local env I can go to as low as 2s without errors and to 1s with ~50% success rate. Initially I wanted to use 20s for starters but then I found multiple places in the same class that use 2min with the same context (waiting for snapshot to happen) introduced by @dakrone. So I went with the same value for consistency.

[dakrone]

LGTM for timeout increases. I think if it keeps failing we should change the ILM completed step to also include the output of the SLM policy retrieval (since that would hopefully include the state of the last manually executed snapshot)