qemu: Add support for full emulation

[cgwalters]

With this, one can e.g.:
cosa run --qemu-image fedora-coreos-36*s390x.qcow2 --arch s390x

This was surprisingly easy. It's tempting to do some full emulation testing for s390x and ppc64le on x86_64/aarch64.

My immediate motivation however is setting up an environment to debug coreos/rpm-ostree#4146

[HuijingHei]

Agree that it will be more convenient to run emulated multi arch on x86_64 platform. Before making it to run, need to install related qemu-system-*.x86_64 packages and will consume more time to start/run emulation arch.

LGTM and maybe need to add dependency packages when build cosa container.

[coreos-assembler]$ ./build.sh
[coreos-assembler]$ sudo make install

[coreos-assembler]$ cosa run --qemu-image rhcos-412.86.202211161950-0-qemu.ppc64le.qcow2 --arch ppc64le
Error: exec: "qemu-system-ppc64": executable file not found in $PATH
2022-11-17T02:52:24Z cli: exec: "qemu-system-ppc64": executable file not found in $PATH
error: failed to execute cmd-run: exit status 1

# This will install all qemu emulation platforms which includes some that we do not need
[coreos-assembler]$ sudo dnf install qemu
Installed:
  SLOF-20210217-4.git33a7322d.fc36.noarch              edk2-aarch64-20220826gitba0e0e4c6a17-1.fc36.noarch   edk2-arm-20220826gitba0e0e4c6a17-1.fc36.noarch   
  openbios-1:20200725-4.git7f28286.fc36.noarch         qemu-2:6.2.0-16.fc36.x86_64                          qemu-system-aarch64-2:6.2.0-16.fc36.x86_64       
  qemu-system-aarch64-core-2:6.2.0-16.fc36.x86_64      qemu-system-alpha-2:6.2.0-16.fc36.x86_64             qemu-system-alpha-core-2:6.2.0-16.fc36.x86_64    
  qemu-system-arm-2:6.2.0-16.fc36.x86_64               qemu-system-arm-core-2:6.2.0-16.fc36.x86_64          qemu-system-avr-2:6.2.0-16.fc36.x86_64           
  qemu-system-avr-core-2:6.2.0-16.fc36.x86_64          qemu-system-cris-2:6.2.0-16.fc36.x86_64              qemu-system-cris-core-2:6.2.0-16.fc36.x86_64     
  qemu-system-m68k-2:6.2.0-16.fc36.x86_64              qemu-system-m68k-core-2:6.2.0-16.fc36.x86_64         qemu-system-microblaze-2:6.2.0-16.fc36.x86_64    
  qemu-system-microblaze-core-2:6.2.0-16.fc36.x86_64   qemu-system-mips-2:6.2.0-16.fc36.x86_64              qemu-system-mips-core-2:6.2.0-16.fc36.x86_64     
  qemu-system-nios2-2:6.2.0-16.fc36.x86_64             qemu-system-nios2-core-2:6.2.0-16.fc36.x86_64        qemu-system-or1k-2:6.2.0-16.fc36.x86_64          
  qemu-system-or1k-core-2:6.2.0-16.fc36.x86_64         qemu-system-ppc-2:6.2.0-16.fc36.x86_64               qemu-system-ppc-core-2:6.2.0-16.fc36.x86_64      
  qemu-system-riscv-2:6.2.0-16.fc36.x86_64             qemu-system-riscv-core-2:6.2.0-16.fc36.x86_64        qemu-system-rx-2:6.2.0-16.fc36.x86_64            
  qemu-system-rx-core-2:6.2.0-16.fc36.x86_64           qemu-system-s390x-2:6.2.0-16.fc36.x86_64             qemu-system-s390x-core-2:6.2.0-16.fc36.x86_64    
  qemu-system-sh4-2:6.2.0-16.fc36.x86_64               qemu-system-sh4-core-2:6.2.0-16.fc36.x86_64          qemu-system-sparc-2:6.2.0-16.fc36.x86_64         
  qemu-system-sparc-core-2:6.2.0-16.fc36.x86_64        qemu-system-tricore-2:6.2.0-16.fc36.x86_64           qemu-system-tricore-core-2:6.2.0-16.fc36.x86_64  
  qemu-system-xtensa-2:6.2.0-16.fc36.x86_64            qemu-system-xtensa-core-2:6.2.0-16.fc36.x86_64       qemu-tools-2:6.2.0-16.fc36.x86_64                
  qemu-user-2:6.2.0-16.fc36.x86_64

[coreos-assembler]$ cosa run --qemu-image rhcos-412.86.202211161950-0-qemu.ppc64le.qcow2 -m 4096 --arch ppc64le
qemu-system-ppc64: warning: TCG doesn't support requested feature, cap-cfpc=workaround
qemu-system-ppc64: warning: TCG doesn't support requested feature, cap-sbbc=workaround
qemu-system-ppc64: warning: TCG doesn't support requested feature, cap-ibs=workaround
qemu-system-ppc64: warning: TCG doesn't support requested feature, cap-ccf-assist=on
Red Hat Enterprise Linux CoreOS 412.86.202211161950-0
  Part of OpenShift 4.12, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.12/architecture/architecture-rhcos.html

---
Last login: Thu Nov 17 03:08:16 2022
[core@cosa-devsh ~]$ arch
ppc64le

[cgwalters]

Ah yes, thanks; forgotten I'd installed all the arch emulators in my dev container. Added those dependencies!

[travier]

Looks fine for me. Wondering how much bigger this will make the COSA container

[jlebon]

Also looks sane to me!

[dustymabe]

Looks fine for me. Wondering how much bigger this will make the COSA container

I'm also wondering about how much bigger this will make the image. Possible improvements there:

• instruct the user to install them when this functionality is attempted to be used.
• OR just limit installing the extra deps on x86_64
  • I imagine it's unlikely people would be trying to emulate another arch on say s390x.

[dustymabe]

I'd like to get to a point where it's really easy for our devs to just grab a session on a multi-arch builder, but until that happens this is a good alternative. Thanks for working on it.

[travier]

I like the "add this only for x86" option as that won't make other arches images bigger. If it's just for a couple MB however, it might not be worth it.

[cgwalters]

OR just limit installing the extra deps on x86_64
I imagine it's unlikely people would be trying to emulate another arch on say s390x.

I think availability of aarch64 is also much greater than ppc64le and s390x - specifically, I can definitely see some people using aarch64 for local workstations. So it's more "emulate ppc64le|s390x on one of aarch64|x86_64" which feels kind of odd to explicitly codify versus just installing all architecture emulators everywhere.

Sizes are

[walters@xenon ~]$ podman run --rm -ti quay.io/coreos-assembler/coreos-assembler shell
[coreos-assembler]$ sudo su -
[root@b4ac0f2818b2 ~]# dnf install qemu-system-{ppc,aarch64,s390x}-core
Repository 'f37-coreos-continuous' is missing name in configuration, using id.
f37-coreos-continuous                                                                                                                               221 kB/s |  44 kB     00:00    
Fedora 37 - x86_64                                                                                                                                   43 MB/s |  64 MB     00:01    
Fedora 37 openh264 (From Cisco) - x86_64                                                                                                            5.9 kB/s | 2.5 kB     00:00    
Fedora Modular 37 - x86_64                                                                                                                          2.9 MB/s | 3.0 MB     00:01    
Fedora 37 - x86_64 - Updates                                                                                                                         12 MB/s |  16 MB     00:01    
Fedora Modular 37 - x86_64 - Updates                                                                                                                8.6 MB/s | 3.7 MB     00:00    
Dependencies resolved.
====================================================================================================================================================================================
 Package                                           Architecture                    Version                                                   Repository                        Size
====================================================================================================================================================================================
Installing:
 qemu-system-aarch64-core                          x86_64                          2:7.0.0-11.fc37                                           updates                          4.8 M
 qemu-system-ppc-core                              x86_64                          2:7.0.0-11.fc37                                           updates                          8.3 M
 qemu-system-s390x-core                            x86_64                          2:7.0.0-11.fc37                                           updates                          2.8 M
Installing dependencies:
 SLOF                                              noarch                          20210217-5.git33a7322d.fc37                               fedora                           218 k
 edk2-aarch64                                      noarch                          20220826gitba0e0e4c6a17-1.fc37                            fedora                           5.6 M
 openbios                                          noarch                          1:20200725-5.git7f28286.fc37                              fedora                           303 k

Transaction Summary
====================================================================================================================================================================================
Install  6 Packages

Total download size: 22 M
Installed size: 271 M
Is this ok [y/N]: 

It looks like edk2-aarch64 expands a lot when uncompressed because there seem to be 3 slightly tweaked 64MB firmware files. But we're mainly concerned about compressed size.

The RPM layer in cosa is 2.48GB right now, so the new (re-compressed) data is just an additional 1% which seems negligible.

My opinion here is that rather than trying to shave a few bits here and there, what we want to do is use rpm-ostree compose image to make the coreos-assembler base layer from RPMs, which will bring all the same benefits around content-addressed sharing. (But we want coreos/rpm-ostree#4012 to really benefit)

[cgwalters]

Two different CI flakes:

• useradd tests/basic: useradd: lock /etc/group.lock already used by PID 4 fedora-coreos-tracker#1250
• fedora infra 503 in mirrorlists (seems to be popping up a lot more)

[dustymabe]

I'm fine with the rationale in #3201 (comment). Leaving it up to @travier and @jlebon on the code review front since they already reviewed it deeper than I have.