[Flaky Test] os/kola/aws, arm: cl.ignition.{v2,v1}.btrfsroot

[sayanchowdhury]

Which tests are flaking?

• cl.ignition.v1.btrfsroot
• cl.ignition.v2.btrfsroot

Description

The test flakes on AWS-ARM and passes on subsequent runs. In the #618 release, the test has not passed even after a couple of runs in Beta but passed on Stable. On Alpha, cl.ignition.v2.btrfsroot on the second run, but the other test kept failing.

Log from the tests (if any)

The instance freezes after throwing the given log in the console

EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

[jepio]

This issue is related: coreos/fedora-coreos-tracker#920.
The first thing wrong with the test is that we don't actually gather console output because we have the parameter set to console=ttyAMA0, while journal.txt of booting instances mentions that the console is ttyS0.

[jepio]

This test also fails on equinix metal analysis shows the following:

• first boot fails at ignition phase with:

[   12.992849] ignition[1317]: disks: createFilesystems: op(3): [finished] creating "btrfs" filesystem on "/run/ignition/dev_aliases/dev/disk/by-partlabel/ROOT"
[   13.008097] ignition[1317]: disks: op(4): [started]  waiting for udev to settle
[   19.471294] systemd-fsck[1343]: Failed to stat /dev/disk/by-label/ROOT: No such file or directory

this blocks the boot, waiting at emergency shell phase for 5 minutes.

• after restart, ignition still needs to run but fails with:

[   16.116745] ignition[1118]: Ignition failed: failed to create filesystems: mkfs failed: exit status 1: Cmd: "/usr/sbin/mkfs.btrfs" "--label=ROOT" "--uuid=9aa5237a-ab6b-458b-a7e8-f25e2baef1a3" "--force" "/run/ignition/dev_aliases/dev/disk/by-partlabel/ROOT" Stdout: "btrfs-progs v5.15.1 \nSee http://btrfs.wiki.kernel.org for more information.\n\n" Stderr: "ERROR: non-unique UUID: 9aa5237a-ab6b-458b-a7e8-f25e2baef1a3\n"

Ignition has this nice comment right before udev settle that nicely explains that this kind of thing is unlikely but can happen: https://github.com/flatcar-linux/ignition/blob/flatcar-master/internal/exec/stages/disks/disks.go#L92-L115.

[pothos]

Yeah, these kind of races are common when modifying block devices. A udev settle means really nothing and only a proper waiter can tell whether the condition we wait for is reached or not.

[pothos]

One way out here would be a waiter directly after the mkfs call which checks that the new filesystem is up under /dev/disk/by-uuid/VALUE based on the expected new UUID. Another solution is to issue a tagged udev event when all mkfs actions are done and wait for it to be reported back, meaning the event queue got completely processed up to this point.

[pothos]

I think it should work to replace the udevadm settle with a udevadm trigger -w /dev/BLOCKDEV directly after the mkfs call.

[pothos]

Tried my luck here: flatcar/ignition#35

[pothos]

Please reopen if you see this race again