kola qemuexec "hangs" during cosa build on apple silicon aarch64

[baude]

Bug Report

Environment

What operating system is being used to run coreos-assembler?
F38 - aarch64

What operating system is being assembled?

FCOS 38

Is coreos-assembler running in Podman or Docker?

podman

If Podman, is coreos-assembler running privileged or unprivileged?

podman run --rm -ti --security-opt=label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap=1001:1001:64536 -v=/home/baude/fcos:/srv/ --device=/dev/kvm --device=/dev/fuse --tmpfs=/tmp -v=/var/tmp:/var/tmp --name=cosa quay.io/coreos-assembler/coreos-assembler:latest shell

Expected Behavior

Actual Behavior

During cosa build on a apple silicon machine having booted Linux (not virtualized), I am seeing failures where kola qemuexec is "hanging". strace reveals it is in a futex/poll loop. when i control-c the build, a TPM is spit out. I am not sure if the TPM error is a result of control-c or is more related to the cause.

shim-aa64 grub2-efi-aa64'
+ supermin --prepare --use-installed -o /srv/tmp/build.qemu/supermin.prepare bash vim-minimal coreutils util-linux procps-ng kmod kernel-modules rpm-ostree distribution-gpg-keys jq systemd dhcp-client iproute selinux-policy selinux-policy-targeted policycoreutils python3 python3-gobject-base buildah podman skopeo iptables iptables-libs python3-pyyaml python3-botocore python3-flufl-lock python3-tenacity cryptsetup gdisk xfsprogs e2fsprogs dosfstools btrfs-progs ca-certificates tar podman grub2 grub2-tools-extra bootupd shim-aa64 grub2-efi-aa64
+ find /usr/lib/coreos-assembler/ -type f
+ find /etc/pki/rpm-gpg/ -type f
+ cat
++ id -u
++ cat /usr/lib/coreos-assembler/supermin-init-prelude.sh
+ chmod a+x /srv/tmp/build.qemu/supermin.prepare/init
+ cd /srv/tmp/build.qemu/supermin.prepare
+ tar -czf init.tar.gz --remove-files init
+ supermin --build /srv/tmp/build.qemu/supermin.prepare --size 10G -f ext2 -o /srv/tmp/build.qemu/supermin.build
++ blkid --output=value --match-tag=UUID /srv/tmp/build.qemu/supermin.build/root
+ superminrootfsuuid=4888e39c-d49d-41de-98f8-66d81c9ecff2
+ echo 'umask 0022'
+ for arg in "$@"
+ printf '%q ' /usr/lib/coreos-assembler/create_disk.sh
+ for arg in "$@"
+ printf '%q ' --config
+ for arg in "$@"
+ printf '%q ' /srv/tmp/build.qemu/image-for-disk.json
+ for arg in "$@"
+ printf '%q ' --kargs
+ for arg in "$@"
+ printf '%q ' 'mitigations=auto,nosmt ignition.config.url=http://192.168.64.1/podman-machine-default.ign ignition.platform.id=qemu'
+ for arg in "$@"
+ printf '%q ' --platform
+ for arg in "$@"
+ printf '%q ' qemu
+ for arg in "$@"
+ printf '%q ' --platforms-json
+ for arg in "$@"
+ printf '%q ' /srv/tmp/platforms.json
+ touch /srv/tmp/build.qemu/runvm-console.txt
+ memory_default=2048
+ case $arch in
+ kola_args=(kola qemuexec -m "${COSA_SUPERMIN_MEMORY:-${memory_default}}" --auto-cpus -U --workdir none --console-to-file "${runvm_console}")
+ base_qemu_args=(-drive 'if=none,id=root,format=raw,snapshot=on,file='"${vmbuilddir}"'/root,index=1' -device 'virtio-blk,drive=root' -kernel "${vmbuilddir}/kernel" -initrd "${vmbuilddir}/initrd" -no-reboot -nodefaults -device virtio-serial -virtfs 'local,id=workdir,path='"${workdir}"',security_model=none,mount_tag=workdir' -append "root=UUID=${superminrootfsuuid} console=${DEFAULT_TERMINAL} selinux=1 enforcing=0 autorelabel=1")
+ for maybe_symlink in "${workdir}"/{src/config,src/yumrepos,builds}
+ '[' -L /srv/src/config ']'
+ for maybe_symlink in "${workdir}"/{src/config,src/yumrepos,builds}
+ '[' -L /srv/src/yumrepos ']'
+ for maybe_symlink in "${workdir}"/{src/config,src/yumrepos,builds}
+ '[' -L /srv/builds ']'
+ '[' -z '' ']'
+ kola qemuexec -m 2048 --auto-cpus -U --workdir none --console-to-file /srv/tmp/build.qemu/runvm-console.txt -- -drive if=none,id=root,format=raw,snapshot=on,file=/srv/tmp/build.qemu/supermin.build/root,index=1 -device virtio-blk,drive=root -kernel /srv/tmp/build.qemu/supermin.build/kernel -initrd /srv/tmp/build.qemu/supermin.build/initrd -no-reboot -nodefaults -device virtio-serial -virtfs local,id=workdir,path=/srv,security_model=none,mount_tag=workdir -append 'root=UUID=4888e39c-d49d-41de-98f8-66d81c9ecff2 console=ttyAMA0 selinux=1 enforcing=0 autorelabel=1' -device virtserialport,chardev=virtioserial0,name=cosa-cmdout -chardev stdio,id=virtioserial0 -drive if=none,id=target,format=qcow2,file=/srv/tmp/build.qemu/fedora-coreos-38.20230430.dev.0-qemu.aarch64.qcow2.tmp,cache=unsafe -device virtio-blk,serial=target,drive=target
^Cqemu-system-aarch64: terminating on signal 2

[coreos-assembler]$ qemu-system-aarch64: tpm-emulator: Could not cleanly shutdown the TPM: Invalid argument
++ rm -rf /srv/builds/38.20230430.dev.0/aarch64/.qemu.building

Reproduction Steps

1. run regular cosa workflow (init, fetch, build)
2. observe failure

Other Information

[cgwalters]

Can you try to use e.g. strace -f -s 2048 -o /tmp/strace.log kola qemuexec ... ?

[baude]

Also prob relevant?

$ cat /srv/tmp/build.qemu/runvm-console.txt 
Synchronous Exception at 0x122CC
Synchronous Exception at 0x0

[baude]

strace.log

[baude]

ok, good news, this can be trivially fixed by upgrading the cosa image to F38. Feel free to close when the team decides to do that?

[baude]

not so fast, it now fails when creating a raw image (but the qcow2 is created correctly)

[cgwalters]

Presumably this is a nested virt issue, right?

It may be worth sanity checking that qemu can be invoked directly. For example, sanity checking with the fcos qemu instructions but really any other OS should work too.

[baude]

Not nested

[dustymabe]

ok, good news, this can be trivially fixed by upgrading the cosa image to F38. Feel free to close when the team decides to do that?

• Bump to Fedora 38 #3451

Merged recently.