Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto re-enable tests #2368

Open
llxia opened this issue Mar 10, 2021 · 1 comment
Open

Auto re-enable tests #2368

llxia opened this issue Mar 10, 2021 · 1 comment

Comments

@llxia
Copy link
Contributor

llxia commented Mar 10, 2021

With a standard format for disabling tests, I think we can create a workflow to auto re-enable tests as mentioned in #1853 (comment)

Scenario 1:
An issue is closed, but the test is still disabled.
(can run on schedule. i.e., weekly or monthly)

  • scan the playlist to find <disable> tag
  • check git issue link to see if the issue is closed
  • if the issue is closed, run this disabled test. If the intermittent label exists, run multiple times. We should be able to get JDK version and platforms based on the label. If not, the information should be in the description. Or just run all available combinations.
  • if the test fails, reopen the issue with detail. Otherwise, create a PR to re-enable the test and remove test excluded label on the issue

Scenario 2:
An issue may already be fixed by other PRs and we want to auto verify and re-enable the test.
This is very similar to Scenario 1. Instead of running disabled tests only if the issue is closed, run the disabled test regardless. If the test passes with different combinations, leave a comment on the issue and create a PR to re-enable the test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Todo
Development

No branches or pull requests

2 participants