shim-x64-15.8-2 update causes secureboot test failures

[dustymabe]

i.e. cosa kola run basic --qemu-firmware=uefi-secure will fail. When I look at the logs all I see is:

Verification failed: (0x1A) Security Violation

What is weird is this update was submitted and got to stable in F38 but not F39. It's in F38 and rawhide and nowhere else:

[dustymabe@hattop ~]$ koji latest-build f39-updates shim
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------
shim-15.6-2                               f39                   rharwood
[dustymabe@hattop ~]$ koji latest-build f38-updates shim
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------
shim-15.8-2                               f38-updates           pjones
[dustymabe@hattop ~]$ koji latest-build f41 shim
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------
shim-15.8-2                               f41                   pjones
[dustymabe@hattop ~]$ koji latest-build f40 shim
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------
shim-15.6-2                               f40                   rharwood

[dustymabe]

pin PR for now: coreos/fedora-coreos-config#2904

[AdamWill]

as the signing process makes shim builds 'special', we just do them for one release then manually tag them to the others. that's ongoing. see https://pagure.io/releng/issue/12018 for details.

I haven't tried doing an SB boot with the new one yet, will try it in a bit.

[AdamWill]

Works fine for me on F40:

[root@xps13a adamw]# rpm -q shim-x64
shim-x64-15.8-2.x86_64
[root@xps13a adamw]# rpm -V shim-x64
[root@xps13a adamw]# mokutil --sb-state
SecureBoot enabled

also, the Beta 1.7 candidate everything netinst has the same shim, and it boots successfully on my test box with SB enabled.

[vathpela]

Can you boot into the working one and attach /sys/firmware/efi/efivars/db-d719b2cb-3d3a-4596-a3bc-dad00e67656f here?

[vathpela]

okay, so... (assuming you're using libvirt), edit the xml description of your vm:

1. boot the VM with a working shim and run mokutil --set-verbosity true, and shut down the VM
2. change the top-level xml entity from <domain type='kvm'> to <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
3. add a section inside <devices> that looks like:

    <serial type='file'>
      <source path='/tmp/qemu.out.log'/>
      <target type='isa-serial' port='1'>
        <model name='isa-serial'/>
      </target>
    </serial>

4. Add a section after </devices> like:

  <qemu:commandline>
    <qemu:arg value='-global'/>
    <qemu:arg value='isa-debugcon.iobase=0x402'/>
    <qemu:arg value='-debugcon'/>
    <qemu:arg value='file:/tmp/qemu.err.log'/>
  </qemu:commandline>

5. boot the non-working shim, stop the VM after the failure, and attach /tmp/qemu.out.log and /tmp/qemu.err.log here.
6. Note that these files can get pretty big on some systems and /tmp is probably tmpfs so they take up RAM, so you want to remove them / disable them pretty soon.

[vathpela]

Do you know what grub version is installed?

[AdamWill]

It might be possible to use /var/tmp instead of /tmp, /var/tmp is disk-backed.

[dustymabe]

/sys/firmware/efi/efivars/db-d719b2cb-3d3a-4596-a3bc-dad00e67656f

attached at: db-d719b2cb-3d3a-4596-a3bc-dad00e67656f.gz

[dustymabe]

Do you know what grub version is installed?

[core@cosa-devsh ~]$ rpm -qa | grep -i grub
grub2-common-2.06-119.fc41.noarch
grub2-pc-modules-2.06-119.fc41.noarch
grub2-tools-minimal-2.06-119.fc41.x86_64
grub2-tools-2.06-119.fc41.x86_64
grub2-efi-x64-2.06-119.fc41.x86_64
grub2-pc-2.06-119.fc41.x86_64

[vathpela]

Can you tar up /sys/firmware/efi/mok-variables/ as well?

[dustymabe]

/sys/firmware/efi/mok-variables/

here: mok-variables.tar.gz

[dustymabe]

5. boot the non-working shim, stop the VM after the failure, and attach /tmp/qemu.out.log and /tmp/qemu.err.log here.

• qemu.out.log.gz
• qemu.err.log.gz

[vathpela]

Oh, I see the problem. I'll have another build for you to test in a few minutes.

[AdamWill]

@vathpela how bad of a problem is it? bad enough to warrant a fresh f40 beta compose?

[vathpela]

fbx64/mmx64 got signed with the old set of certs we don't trust any more. So yeah, I think so.

[vathpela]

https://koji.fedoraproject.org/koji/taskinfo?taskID=115176363 should have them signed correctly.

[AdamWill]

ah, great :/ that's the magic fallback path thing, right? so fallback path recovery for deployed systems is what's broken? and that can affect things like cloud environments where we just expect to boot from fallback path, I guess...

[dustymabe]

shim-x64-15.8-3.x86_64 from https://koji.fedoraproject.org/koji/taskinfo?taskID=115176363 seems to work fine.

how do we make a bodhi update show up for that?

[AdamWill]

since 15.8-2 got pushed stable for F38 I guess Peter needs to create a new one. I've filed https://bugzilla.redhat.com/show_bug.cgi?id=2270355 as a blocker candidate. It would be great if you could add any details or corrections there, @vathpela .

[vathpela]

https://bodhi.fedoraproject.org/updates/FEDORA-2024-d6b824677a

[vathpela]

@vathpela how bad of a problem is it? bad enough to warrant a fresh f40 beta compose?

No, not worth re-doing the beta.

[dustymabe]

All streams now have the new 15.8-3 shim.

Just in case it wasn't clear: this problem was caught in CI and never entered any releases on production streams.