mantle/kola: Auto enable rerun success in some failure scenarios #3494
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This tag will be used when the test framework detects tests failing in specific ways that we've decided we want to allow rerun success for.
This will allow tests that get aborted due to a timeout to not fail the overall run if a rerun is attempted and it passes. One example of where this is useful is if a test times out on initialization and never reaches the machine via SSH (which we see regularly). In that case is the failure an issue with the platform bringing up the machine or a fundamental issue with the software inside FCOS/RHCOS? If it's an issue with the platform and the rerun succeeds then we don't want to see a failure. If it's a fundamental issue with the software inside (i.e. Ignition fails) then the rerun will fail anyway and this will do no harm.
This patch enables us to have console checks that are non-fatal and will print a log message to the screen and nothing more, which was stated as desirable in [1]. This commit also re-works the implementation of the console/journal checks in runTest() to deduplicate code. It has the side-effect of making SkipConsoleWarnings apply to journal checks too, but I think that's actually a benefit and not a negative. [1] coreos#3450 (comment)
This will make it so there are console checks we can define that, if configured, will mark a test as able to have a test run succeed if a rerun succeeds.
Now that we have warnOnly and allowRerunSuccess capabilities in our consoleChecks let's add back in the kernel soft lockup check that was removed in 7283c89.
This function conveniently exists so let's use it.
Since we call runProvidedTests() for both the first run and the rerun let's not call the variable firstRunErr since in the nested call that actually won't be accurate. Let's just call it runErr instead.
In this case it's usually the platform having some internal failure. If it succeeds in the rerun then just be happy with that.
This was referenced May 31, 2023
gursewak1997
approved these changes
May 31, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This will auto enable rerun success for some failure scenarios including timeouts and some console checks.