-
Notifications
You must be signed in to change notification settings - Fork 165
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
coreos.ignition.failure
sometimes fails on RHCOS
#3670
Comments
`coreos.unique.boot.failure` has been failing on aarch64 builds and `coreos.ignition.failure` has been failing on RHCOS in general. Let's denylist them and add a snooze while we investigate the issues. See: coreos/coreos-assembler#3670 and coreos/coreos-assembler#3669
I ran a simple script locally which tested the coreos.ignition.failure 100 times. Not once was I able to reproduce this failure.
|
Since I couldn't reproduce this locally, I will try to un-denylist the test and watch it fail on the pipeline, to investigate the logs and better understand the underlying issue on RHCOS 4.16. When the logs are available, I will once more add this test to the denylist. |
coreos.ignition.failure has been failing on RHCOS in general. After failing to reproduce it locally I am removing it from denylist in order to catch errors in the pipeline. The goal of this PR is to allow it in RHCOS 4.16, catch the errors from the log and re-introduce `coreos.ignition.failure` to kola-denylist while working on solution. See: coreos/coreos-assembler#3670 and coreos/coreos-assembler#3669
After failing to reproduce `coreos.ignition.failure` locally I am removing it from denylist in order to catch errors in the pipeline. The goal of this PR is to allow it in RHCOS 4.16, catch the errors from the log and re-introduce `coreos.ignition.failure` to kola-denylist while working on solution. See: coreos/coreos-assembler#3670
After failing to reproduce `coreos.ignition.failure` locally I am removing it from denylist in order to catch errors in the pipeline. The goal of this PR is to allow it in RHCOS 4.16, catch the errors from the log and re-introduce `coreos.ignition.failure` to kola-denylist while working on solution. See: coreos/coreos-assembler#3670
Does this happen on x86_64 or one of the secondary arches? If x86_64 you can check the If multi-arch did you try in a debug pod? |
I only came across a reference to this issue on aarch64. |
@marmijo comments on the matter:
|
This test is still failing so let's extend the snooze for a few weeks so we can continue to investigate coreos/coreos-assembler#3670
Right now I am running a Jenkins debug-pod with rhcos [4.16-9.2] on aarch64 - attempting to reproduce the FAIL for coreos.ignition.failure in a loop:
As of the first many runs, no failures were detected. I will attempt the same on ppc64le and the s390x. |
The same as above goes for the ppc64le:
|
The other thing you could do here is keep the denylist entry but
Yeah, that makes sense. The reason is that we're currently keeping the Let's fix the console issue first before we turn it back on via |
…lure See: #3670 Reference: #3670 (comment)
Just for clarity, this is already implemented in here. |
Today a failure was observed on 4.12 aarch64 412.86.202402141419-0.
|
Additionally console.txt from the jenkins job |
So the test failed once but then passed in the rerun.
Is that log from the failed run or from the successful rerun? I honestly think this was just some sort of infra flake. |
Yeah, I wouldn't be worried about that (well, maybe we can strengthen our handling of the QMP socket). Note also it's for 4.12, which doesn't have the changes that precipitated this issue. |
I first thought this log was from the failed test, but it's from the successful rerun (lesson learned).
Noted, thanks for the comments - sorry for the noise. |
Failure related to this test wasn't observed in a few weeks. See: coreos/coreos-assembler#3670
The problem related to this issue wasn't observed in a while. I failed to reproduce it both locally and in the pipeline. |
coreos.ignition.failure
on RHCOS sometimes passes (like it did originally in #3647) and sometimes fails with:like in #3668.
The text was updated successfully, but these errors were encountered: