Rerun failed tests #126
Merged
Rerun failed tests #126
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dnephin
force-pushed
the
rerun-fails
branch
6 times, most recently
from
6b4f95b
to
16e5df2
Compare
dnephin
force-pushed
the
rerun-fails
branch
4 times, most recently
from
695fc6d
to
0c589cf
Compare
dnephin
commented
Jun 8, 2020
This fixes a bug, and prevents one in the new rerun-fails feature. Incomplete tests were printed in the summary, but never passed to the EventHandler. Now when the execution ends, fake TestEVent will be created and sent to the Handler, so that it can print the output of the failed test. This allows the rerun-fails loop to include these failed tests in the count, and avoid a situation where a test was panicing, but not getting rerun. If the panic test did not get rerun then 'go test' may exit 0, making it look like everything had passed, when in fact one test was still panicing. This type of bug sounds pretty severe.
Also print help text to stdout when it is requested with --help. Continue to print to stderr when it is a result of an invalid flag. Also add missing command help to root command.
Rename the rerun-fails flag, and improve the help text. Also add a test for options.Validate
Use -count=1 because there appears to be a bug in the 'go test' cache that has it PASS (cached) instead of failing when SEED=3.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a new
--rerun-fails[=n]flag. When the flag is usedgotestsumwill rerun any failed tests up tontimes until each one passes at least once.The new feature can be used to mitigate the pain of flaky tests in CI. To avoid unnecessarily rerunning tests when there is a bug or problem with the test environment, if there are more than 10 failed tests, rerun will be skipped. The
--rerun-fails-max-failuresflag can be used to change that threshold.Using the
--rerun-failsflag along with--raw-commandor withgo testargs may require special handling, which will need to be documented in #117