kola/kernel-replace: adapt for RHCOS and SCOS

[c4rt0]

There were instances where kola/switch-kernel test would fail.
It was recommended to remove it (PR) and generalize this test, so that it can work with FCOS, RHCOS and SCOS.

[jbtrystram]

Shellcheck will complain other than this it looks good.
Could you run the test locally on RHCOS ?

[c4rt0]

Could you run the test locally on RHCOS ?

Just tried it now, it's failing. I'm now investigating:

=== RUN   ext.config.shared.rpm-ostree.kernel-replace
systemctl status kola-runext.service:
× kola-runext.service
     Loaded: loaded (/etc/systemd/system/kola-runext.service; static)
     Active: failed (Result: exit-code) since Fri 2024-07-05 18:34:49 UTC; 399ms ago
   Duration: 36.717s
    Process: 1266 ExecStart=/usr/local/bin/kola-runext-kernel-replace (code=exited, status=1/FAILURE)
   Main PID: 1266 (code=exited, status=1/FAILURE)
        CPU: 33.418s

Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1270]: ++ grep '^ID=' /etc/os-release
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + OS_ID=rhcos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + image_dir=/var/tmp/coreos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + image=oci:/var/tmp/coreos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + image_pull=ostree-unverified-image:oci:/var/tmp/coreos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + tmp_imagedir=/var/tmp/coreos-tmp
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1273]: ++ arch
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + arch=x86_64
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + kver=6.2.9-300.fc38.x86_64
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + case "${AUTOPKGTEST_REBOOT_MARK:-}" in
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + skopeo copy oci:/var/tmp/coreos containers-storage:localhost/rhcos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1274]: time="2024-07-05T18:34:13Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1274]: Getting image source signatures
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1274]: Copying blob sha256:5d033c3086bc3b8a75fe007c50fa48d18cdea48560bd6d6b2b359d79397bb167
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1274]: Copying config sha256:5b8ccce2c709b5c14f75df3761da4eccfcef89d39165711f14c71196df75b28c
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1274]: Writing manifest to image destination
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + rm /var/tmp/coreos -rf
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1307]: ++ mktemp -d
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + td=/tmp/tmp.q2EfPgCVdW
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + cd /tmp/tmp.q2EfPgCVdW
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1310]: ++ grep Version
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1309]: ++ rpm-ostree --version
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + version=' Version: '\''2024.3'\'''
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + cat
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + touched_resolv_conf=0
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + test '!' -f /etc/resolv.conf
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + podman build --net=host -t localhost/rhcos-derived --squash .
Jul 05 18:34:48 qemu0 podman[1313]: 2024-07-05 18:34:48.253248775 +0000 UTC m=+0.050040248 system refresh
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1313]: STEP 1/3: FROM localhost/rhcos
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1313]: STEP 2/3: RUN rpm-ostree cliwrap install-to-root /
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1345]: Successfully enabled cliwrap for /
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1313]: STEP 3/3: RUN rpm-ostree override replace     https://koji.fedoraproject.org/koji/buildinfo?buildID=2178613 &&     ostree container commit
Jul 05 18:34:49 qemu0 kola-runext-kernel-replace[1397]: Downloading https://koji.fedoraproject.org/koji/buildinfo?buildID=2178613...done
Jul 05 18:34:49 qemu0 kola-runext-kernel-replace[1397]: (rpm-ostree override replace:2): libdnf-WARNING **: 18:34:49.701: failed to read RPM: /tmp/.tmpysF0mq/13.rpm: not a rpm, skipping
Jul 05 18:34:49 qemu0 kola-runext-kernel-replace[1397]: error: Invalid RPM file: /tmp/.tmpysF0mq/13.rpm
Jul 05 18:34:49 qemu0 podman[1313]: 2024-07-05 18:34:49.787637333 +0000 UTC m=+1.584428810 image build  
Jul 05 18:34:49 qemu0 kola-runext-kernel-replace[1313]: Error: building at STEP "RUN rpm-ostree override replace     https://koji.fedoraproject.org/koji/buildinfo?buildID=2178613 &&     ostree container commit": while running runtime: exit status 1
Jul 05 18:34:49 qemu0 systemd[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
Jul 05 18:34:49 qemu0 systemd[1]: kola-runext.service: Failed with result 'exit-code'.
Jul 05 18:34:49 qemu0 systemd[1]: kola-runext.service: Consumed 33.418s CPU time.
--- FAIL: ext.config.shared.rpm-ostree.kernel-replace (171.99s)
        cluster.go:162: Error: Unit kola-runext.service exited with code 1
        cluster.go:162: 2024-07-05T18:34:50Z cli: Unit kola-runext.service exited with code 1
        harness.go:1265: kolet failed: : kolet run-test-unit failed: Process exited with status 1
FAIL, output in tmp/kola
Error: harness: test suite failed

[jlebon]

Trying to replace the RHCOS kernel with a Fedora one in theory should work, but I think we can get closer to what users are likely to do by replacing with a CentOS Stream kernel instead. So basically:

• FCOS: replace with Fedora kernel
• RHCOS and SCOS: replace with CentOS Stream kernel

For Fedora, the test currently pins to a specific Koji build, which is stale now. We should bump it. Once we have dnf in all the streams, we should probably change that actually to just dynamically always take the latest kernel that's != to the kernel embedded in the container image (similarly for RHCOS/SCOS too).

[c4rt0]

I ran this on both rhcos (4.17) and the latest fcos build. Test passes as expected.

[c4rt0]

Trying to replace the RHCOS kernel with a Fedora one in theory should work, but I think we can get closer to what users are likely to do by replacing with a CentOS Stream kernel instead. So basically:

* FCOS: replace with Fedora kernel

* RHCOS and SCOS: replace with CentOS Stream kernel

That's now done.

For Fedora, the test currently pins to a specific Koji build, which is stale now. We should bump it.

That's also done.

Once we have dnf in all the streams, we should probably change that actually to just dynamically always take the latest kernel that's != to the kernel embedded in the container image (similarly for RHCOS/SCOS too).

I can look into this in the future.

[c4rt0]

Again, latest changes successfully passed test both on fcos and rhcos. Thank you for all of the suggestions!

Completed.

[jlebon]

Suggested commit title:

kola/kernel-replace: adapt for RHCOS and SCOS

For the commit message, I think in general we should avoid just linking to some other URL that has all the context. Let's inline that context, and provide the URL only as an additional reference in the footer.

[jlebon]

LGTM overall. Thoughts on suggested commit title in #3048 (review) ? I think mentioning OS_ID there isn't really capturing it.

[jbtrystram]

LGTM overall. I am a bit concerned by the hard-coded kernel package URLs, won't they go stale eventually, breaking the test?

I don't have a good alternative to suggest though, except adding an item to the rebase checklist..

[c4rt0]

LGTM overall. I am a bit concerned by the hard-coded kernel package URLs, won't they go stale eventually, breaking the test?

I don't have a good alternative to suggest though, except adding an item to the rebase checklist..

I agree, @jlebon also talked about it here: #3048 (comment). Let's wait with that until we have DNF in all of the streams.