Distrobuilder fails to build rockylinux-9 vm flavour

[seingierf]

Hello,

The following command fails while building a rockylinux-9 vm flavour:

sudo distrobuilder build-incus images/rockylinux.yaml -o image.variant=cloud -o image.release=9 -o source.variant=boot --vm --type unified --debug
[...]
INFO   [2024-03-03T11:37:11+01:00] Running hooks                                 trigger=post-files
++ ls /boot/initramfs-5.14.0-362.18.1.el9_3.0.1.x86_64.img
++ sed -r 's#.*initramfs-(.+)\.img#\1#'
+ kver=5.14.0-362.18.1.el9_3.0.1.x86_64
+ target=/boot/efi/EFI/rocky/grub.cfg
+ grub2-mkconfig -o /boot/efi/EFI/rocky/grub.cfg
Generating grub configuration file ...
Adding boot menu entry for UEFI Firmware Settings ...
done
+ sed -i 's#root=[^ ]*#root=/dev/sda2#g' /boot/efi/EFI/rocky/grub.cfg
+ grubby --update-kernel=/boot/vmlinuz-5.14.0-362.18.1.el9_3.0.1.x86_64 '--args=root=/dev/sda2 ro'
The param /boot/vmlinuz-5.14.0-362.18.1.el9_3.0.1.x86_64 is incorrect
Error: Failed to run post-files: exit status 1
ERROR  [2024-03-03T11:37:13+01:00] Failed running distrobuilder                  err="Failed to run post-files: exit status 1"

I patched the yaml file to be slightly more verbose during post-install:

diff --git a/images/rockylinux.yaml b/images/rockylinux.yaml
index 3edb130..c97ebd0 100644
--- a/images/rockylinux.yaml
+++ b/images/rockylinux.yaml
@@ -372,11 +372,19 @@ actions:
     set -eux
     kver=$(ls /boot/initramfs-*.img | sed -r 's#.*initramfs-(.+)\.img#\1#')
     target=/boot/efi/EFI/rocky/grub.cfg
+    grep boot /boot/loader/entries/*.conf
     # Create grub.cfg file
+    sed -i '1s/.*/& -x/' /usr/sbin/grub2-mkconfig
     grub2-mkconfig -o "${target}"
+    cat /boot/efi/EFI/rocky/grub.cfg
     sed -i "s#root=[^ ]*#root=/dev/sda2#g" "${target}"
     # Update files in /boot/loader/entries/. `grubby` needs to be run after
     # `grub2-mkconfig` as the latter overwrites files in /boot/loader/entries/.
+    for cf in /boot/loader/entries/*.conf; do
+      echo "### ${cf}"
+      cat "${cf}"
+    done
+    sed -i '1s/.*/& -x/' /usr/libexec/grubby/grubby-bls
     grubby --update-kernel=/boot/vmlinuz-${kver} --args="root=/dev/sda2 ro"
     # Regenerate initramfs
     dracut --kver "${kver}" -f

See attached log file rockylinux9-vm.log.

[stgraber]

You're not passing the architecture through the command line which is probably why this is falling when it's not on our own infrastructure.

Note that the YAML files here are those used verbatim in production so if you see the image on our image server you can look at the build logs on https://jenkins.linuxcontainers.org to figure out what you're doing differently

[seingierf]

I tried the architecture parameter without success.

I checked the logs on Jenkins and noticed that the image was created in 2 steps: build-dir then pack-incus.
If I run those commands, I get the expected incus.tar.gz + disk.qcow2 files and can import/launch the image successfully.

Back to my initial build-incus attempt, I can work around the aforementioned error by adding the following sed command to the post-files section:

diff --git a/images/rockylinux.yaml b/images/rockylinux.yaml
index 3edb130..2df828d 100644
--- a/images/rockylinux.yaml
+++ b/images/rockylinux.yaml
@@ -372,6 +372,7 @@ actions:
     set -eux
     kver=$(ls /boot/initramfs-*.img | sed -r 's#.*initramfs-(.+)\.img#\1#')
     target=/boot/efi/EFI/rocky/grub.cfg
+    sed -ri 's,/var/cache/distrobuilder[^/]+/rootfs,,' /boot/loader/entries/*.conf
     # Create grub.cfg file
     grub2-mkconfig -o "${target}"
     sed -i "s#root=[^ ]*#root=/dev/sda2#g" "${target}"

[stgraber]

@monstermunchkin any idea why building in one step isn't resulting in the same as the two separate steps here?

[monstermunchkin]

No idea but I'll look into this.

[monstermunchkin]

There's something I noticed. When running build-incus, I see

  Running scriptlet: kernel-core-5.14.0-362.18.1.el9_3.0.1.x86_64                                                                                                         159/159 
grub2-probe: error: failed to get canonical path of `/dev/mapper/nvme0n1p2_crypt'.
No path or device is specified.
Usage: grub2-probe [OPTION...] [OPTION]... [PATH|DEVICE]
Try 'grub2-probe --help' or 'grub2-probe --usage' for more information.
findmnt: can't read (null): No such file or directory
/usr/lib/dracut/dracut-functions.sh: line 775: btrfs: command not found

When running pack-incus, I see

  Running scriptlet: kernel-core-5.14.0-362.18.1.el9_3.0.1.x86_64                                                                                                           68/68 
grub2-probe: error: failed to get canonical path of `overlay'.
No path or device is specified.
Usage: grub2-probe [OPTION...] [OPTION]... [PATH|DEVICE]
Try 'grub2-probe --help' or 'grub2-probe --usage' for more information.
findmnt: can't read (null): No such file or directory

[stgraber]

Is there a way to turn off the grub2-probe stuff? It's been causing issues by trying to scan various host mounts and devices.

[monstermunchkin]

Is there a way to turn off the grub2-probe stuff? It's been causing issues by trying to scan various host mounts and devices.

I haven't found a way of doing this.