Retry for OpenSearch Container

[nhtruong]

Description

Tests that run docker-compose up randomly fail with "no tests found error". Rerunning the failed jobs resolves the issue. This is a bug in Opensearch build that causes the container to intermittently fail. Restarting the container when encountering this issue is a workaround this PR is addressing.

Issues Resolved

closes #268

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dblock]

Maybe we can either fix the underlying issue or use something like https://www.npmjs.com/package/retry instead of rolling our own?

[nhtruong]

@dblock I'm looking into finding/fixing the source myself. I'm just checking if we can use retry to ease the pain of manually retrying failed jobs. Unfortunately, the retry I implemented in the first commit will still fail all 3 attempts if the first attempt fails. Now I'm using this PR to test out different strategies. My current suspicion is that TAP is somehow looking for TS or JSX files as if TAP_TS and TAP_TXS were set to 1.

[nhtruong]

Found the cause and a workaround. Please check the updated description for details.

[dblock]

I hate it but I think it works ;)

[dblock]

Berfore we merge this, is there a way to retry the tests at GHA level? I'm worried about hacks on top of hacks.

[harshavamsi]

Could look into https://github.com/marketplace/actions/retry-action and https://github.com/marketplace/actions/retry-step. Validate container start up docker and execute retry logic.

[nhtruong]

Could look into https://github.com/marketplace/actions/retry-action and https://github.com/marketplace/actions/retry-step. Validate container start up docker and execute retry logic.

I thought of retry-step for the Container Startup step but decided not to use it:

• It will retry the whole step while we only need to restart the container instead of going through the entire clean/build/boot-up sequence.
• This step never failed (or failed silently), so we will still have to modify this step to grep for the kill signals to call exit 1 explicitly to signal retry-step to retry.
• The workaround is in the make file, people running make commands on their local machine will benefit from this too, not just github workflow.

I agree that grepping for a piece of text is not pretty 😞. There are potentially more elegant workarounds if we want to invest more time.

[nhtruong]

Maybe we can use this: https://hub.docker.com/r/willfarrell/autoheal/
I'll take a look this afternoon.

[nhtruong]

@dblock @harshavamsi
Sooo the best the solution is stupidly simple! The docker-compose up step not failing on the failed tests led me to believe that the container was still running and only the OS service inside the container that crashed. However, what actually happened is, the container itself crashed soon after docker-compose up. And we just need to tell docker to always restart the container (instead of the default no) when it crashes.

https://hub.docker.com/r/willfarrell/autoheal/ is not appropriate for this situation because it's mean for crashed services, not crashed containers.

[nhtruong]

Testing a few more times to make sure that the problem is solved.

[nhtruong]

I spoke too soon. This bug doesn't always crash the container when the opensearch processed is killed. It only does that sometimes.

[dblock]

@nhtruong This is pretty clean retry-wise. Do you want to merge it or it doesn't work?

[nhtruong]

@dblock I've tested this on over 700 runs and non failed. So, it's good to go!