Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PROPOSAL] Add flaky test retry #139

Open
Tracked by #1818 ...
dblock opened this issue Apr 6, 2022 · 3 comments
Open
Tracked by #1818 ...

[PROPOSAL] Add flaky test retry #139

dblock opened this issue Apr 6, 2022 · 3 comments
Labels
campaign Parent issues of OpenSearch release campaigns. Meta

Comments

@dblock
Copy link
Member

dblock commented Apr 6, 2022

What kind of business use case are you trying to solve? What are your requirements?
I want to be able to rely on CI when contributing to all plugin projects.

What is the problem? What is preventing you from meeting the requirements?
In a trivial change in opensearch-project/.github#64 I made 20 pull requests with a trivial .md file change. 10/20 have failed in CI, anecdotally because of flakey tests.

What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?
Introduce a small number of retries as standard procedure to tests, similar to opensearch-project/OpenSearch#2547, implemented in opensearch-project/OpenSearch#2638. While this doesn't fix flakey tests, it decreases the noise to a level where the worst offenders can be fixed instead of having a 50% failure rate.

What are remaining open questions?
We know how to do this for OpenSearch plugins with Gradle, but we need another mechanism for OpenSearch Dashboards.

@saratvemulapalli
Copy link
Member

@dblock looks like this slipped through the cracks.
Do you want to open up issues on all the repos and label them for next release?

@dblock
Copy link
Member Author

dblock commented Sep 19, 2022

@saratvemulapalli if you want to take this campaign up go for it!

@ohltyler
Copy link
Member

ohltyler commented Jan 5, 2023

Similar comment as #138 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
campaign Parent issues of OpenSearch release campaigns. Meta
Projects
None yet
Development

No branches or pull requests

4 participants