-
Notifications
You must be signed in to change notification settings - Fork 118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
--rerun-fails do not execute remaining tests after rerun test passes #278
Comments
Thank you for the bug report!
This one might be difficult to fix. Because of some limitations of the You might be able to use the If we can confirm that upstream problems with
This is the expected behaviour of If you want to re-run everything I think maybe It's possible that behaviour could be added to
This one I don't understand yet. From what I can tell it should use the same formatter for everything. Are you sure it's the formatter, and not the summary? I would expect to see the failed test output in the summary at the end of the run (even if the command returns success because the re-runs passed). I would not want to hide this by default, because I think it's important to see that output. If the output is indeed coming from the summary at the end, then I think maybe the change to not using |
It sounds like most likely the for run in {0..3}; do
if gotestsum --format short-verbose --jsonfile=last-run.log --hide-summary=all ... -- -failfast ./...; then
exit 0
fi
done
# print the output for the last run
gotestsum --raw-command --format short-verbose -- cat last-run.log
rm -f last-run.log
exit 1 |
Perhaps I didn't explain properly, I meant to say that remaining 25 unexecuted tests do not run at all after 5th test passes upon rerun.
Test 6 to Test 30 do not even execute at all. |
I will try this today and let you know. |
Ahhh, I understand now. That's because of I think |
#280 adds that error so that Anything else we should address for this issue? Did the suggestion in the comment above work out for your use case? |
I think we've fixed the bug here, so I'm going to close this issue. If there's something I've missed please do leave a comment and I can re-open it. Thanks! |
Hi,
Thank you for your contribution to the
gotestsum
package. It is extremely useful.We started using the
--rerun-fails
flag to rerun potentially unstable test suites to prevent false failures in our CI/CD pipeline.However, we identified 2 issues with this:
--format short-verbose
does not take effect for the failed tests. We get thestandard-verbose
output for the failed test case. As a result console log shows that the test suite has failed, but in reality, the test was passed in the retry.Below is our usage scenario. Each service has n number of tests (Service-1 has 10 tests, Service-2 has 30 tests, and Service-3 has 4 tests).
Here is an example of gotestsum usage in Makefile BEFORE --rerun-fails
Here is an example of gotestsum usage in Makefile AFTER --rerun-fails
We want to achieve the following outcome with respect to our usecase above:
Appreciate your help 🙏
The text was updated successfully, but these errors were encountered: