Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fwupdmgr - UEFI ESP partition not detected or configured #1623

Open
travier opened this issue Nov 29, 2023 · 39 comments
Open

fwupdmgr - UEFI ESP partition not detected or configured #1623

travier opened this issue Nov 29, 2023 · 39 comments

Comments

@travier
Copy link
Member

travier commented Nov 29, 2023

Describe the bug

fwupd is unable to update firmwares as the EFI partition is not mounted anymore in FCOS: #694

See: https://discussion.fedoraproject.org/t/coreos-fwupdmgr-uefi-esp-partition-not-detected-or-configured/97522

Reproduction steps

  1. Install FCOS on a hardware with firmware updates available
  2. Get a message suggesting you update

Expected behavior

Able to update firmwares

Actual behavior

Unable to update firmwares

System details

Bare metal

Butane or Ignition config

N/A

Additional information

@travier
Copy link
Member Author

travier commented Nov 30, 2023

This is tricky. We likely need to add support to fwupd to understand how FCOS is setup to mount the partition on demand.

@hughsie
Copy link

hughsie commented Dec 2, 2023

Hi all. I'm a bit confused what the issue is -- we do mount the EFI partition on demand as needed...

@dustymabe
Copy link
Member

dustymabe commented Dec 6, 2023

even with me manually mounting the ESP under /boot/efi it still reports it can't find anything:

[core@cosa-devsh ~]$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda    252:0    0   10G  0 disk 
├─vda1 252:1    0    1M  0 part 
├─vda2 252:2    0  127M  0 part /boot/efi
├─vda3 252:3    0  384M  0 part /boot
└─vda4 252:4    0  9.5G  0 part /var
                                /sysroot/ostree/deploy/fedora-coreos/var
                                /usr
                                /etc
                                /
                                /sysroot

[core@cosa-devsh ~]$ sudo /usr/bin/fwupdtool esp-list --verbose
16:46:14.524 FuDebug              verbose to info (on console 1)
16:46:14.525 FuEngine             starting fwupd 1.9.9…
No ESP or BDP found

NOTE: this is in a VM so it might be an invalid test?

@hughsie
Copy link

hughsie commented Dec 6, 2023

@dustymabe can you attach udisksctl dump please?

@jlebon
Copy link
Member

jlebon commented Dec 6, 2023

@dustymabe can you attach udisksctl dump please?

Heh, was just about to post related to this. Naively perusing the fwupd source, it seems like fwupd actually relies on UDisks2 being installed and activated but we don't ship udisks2 in FCOS.

Is the expectation that we set e.g. EspLocation ourselves in that case? Can that take a block device or just a mount point?

@jlebon
Copy link
Member

jlebon commented Dec 6, 2023

This is related: coreos/bootupd#132

We also need to support possibly updating multiple ESPs.

@dustymabe
Copy link
Member

@jlebon
Is the expectation that we set e.g. EspLocation ourselves in that case? Can that take a block device or just a mount point?

According to https://github.com/fwupd/fwupd/wiki/PluginFlag:esp-not-found

If you can't solve the problem by installing UDisks you should manually set OverrideESPMountPoint in /etc/fwupd/uefi_capsule.conf (formerly uefi.conf).

@dustymabe
Copy link
Member

though I think that wiki page might be out of date. According to man 5 fwupd.conf it just goes in /etc/fwupd/fwupd.conf under the [fwupd] section. So I think we should end up with:

[fwupd]
EspLocation=/boot/efi

??

@dustymabe
Copy link
Member

dustymabe commented Dec 6, 2023

With /dev/vda2 the daemon fails to start:

[core@cosa-devsh ~]$ sudo cat /etc/fwupd/fwupd.conf 
# use `man 5 fwupd.conf` for documentation
[fwupd]
DisabledPlugins=test;test_ble
OnlyTrusted=true
AllowEmulation=false
EspLocation=/dev/vda2

[core@cosa-devsh ~]$ sudo systemctl stop fwupd
[core@cosa-devsh ~]$ sudo systemctl start fwupd
Job for fwupd.service failed because the control process exited with error code.
See "systemctl status fwupd.service" and "journalctl -xeu fwupd.service" for details.
[core@cosa-devsh ~]$ 
(failed reverse-i-search)`stt': sudo systemctl ^Cart fwupd
[core@cosa-devsh ~]$ systemctl status fwupd.service 
× fwupd.service - Firmware update daemon
     Loaded: loaded (/usr/lib/systemd/system/fwupd.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Wed 2023-12-06 17:57:19 UTC; 7s ago
       Docs: https://fwupd.org/
    Process: 2587 ExecStart=/usr/libexec/fwupd/fwupd (code=exited, status=1/FAILURE)
   Main PID: 2587 (code=exited, status=1/FAILURE)
        CPU: 64ms

Dec 06 17:57:19 cosa-devsh systemd[1]: Starting fwupd.service - Firmware update daemon...
Dec 06 17:57:19 cosa-devsh fwupd[2587]: Failed to load daemon: failed to load engine: failed to call org.freedesktop.UDisks2.Manager.GetBlockDevices(): The name is not activatable
Dec 06 17:57:19 cosa-devsh systemd[1]: fwupd.service: Main process exited, code=exited, status=1/FAILURE
Dec 06 17:57:19 cosa-devsh systemd[1]: fwupd.service: Failed with result 'exit-code'.
Dec 06 17:57:19 cosa-devsh systemd[1]: Failed to start fwupd.service - Firmware update daemon.

but with /boot/efi the daemon doesn't crash, but nothing is improved:

[core@cosa-devsh ~]$ sudo cat /etc/fwupd/fwupd.conf 
# use `man 5 fwupd.conf` for documentation
[fwupd]
DisabledPlugins=test;test_ble
OnlyTrusted=true
AllowEmulation=false
EspLocation=/boot/efi

[core@cosa-devsh ~]$ sudo systemctl stop fwupd
[core@cosa-devsh ~]$ sudo systemctl start fwupd
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ sudo /usr/bin/fwupdtool esp-list --verbose
17:59:02.428 FuDebug              verbose to info (on console 1)
17:59:02.429 FuEngine             starting fwupd 1.9.9…
No ESP or BDP found


@dustymabe
Copy link
Member

We discussed this in the community meeting today:

│11:50:08* dustymabe | !info this one needs more investigation and we're looking for a volunteer to dig in

@dustymabe dustymabe removed the meeting topics for meetings label Dec 6, 2023
@hughsie
Copy link

hughsie commented Dec 7, 2023

We also need to support possibly updating multiple ESPs.

Then it either needs to be hardcoded or you need to use udisks so that we can use a heuristic to guess the correct device. e.g. ignore Recovery Partition, prefer ESPs with linux files on, and prefer the ESP flag set over msftdata. I don't know how you'd do all that with raw unix APIs.

fwupd actually relies on UDisks2 being installed and activated but we don't ship udisks2 in FCOS

So how is fwupd supposed to mount the partition on demand? EspLocation=/boot/efi is only going to work if the partition is already mounted. I don't mind using something other than udisks for FCOS, but shelling out to /usr/bin/mount is pretty janky at best...

@hughsie
Copy link

hughsie commented Dec 7, 2023

even with me manually mounting the ESP

can you do something like sudo parted -l /dev/nvme0n1 -- you should see something like

Number  Start   End     Size    File system  Name                  Flags
 1      1049kB  630MB   629MB   fat32        EFI System Partition  boot, esp
 2      630MB   1704MB  1074MB  ext4
 3      1704MB  76.9GB  75.2GB  ext4
 4      76.9GB  256GB   179GB

@superm1
Copy link

superm1 commented Dec 7, 2023

Maybe a dumb question; but why can't FCOS add udisks2? There is a real use case here for mounting a partition dynamically. Isn't the space trade off worth it?

@dustymabe
Copy link
Member

even with me manually mounting the ESP

can you do something like sudo parted -l /dev/nvme0n1

We don't have parted by default in FCOS:

[core@cosa-devsh ~]$ sudo fdisk -l /dev/vda                                                                                                                                                  
Disk /dev/vda: 10 GiB, 10737418240 bytes, 20971520 sectors                                                                                                                                   
Units: sectors of 1 * 512 = 512 bytes                                                                                                                                                        
Sector size (logical/physical): 512 bytes / 512 bytes                                                                                                                                        
I/O size (minimum/optimal): 512 bytes / 512 bytes                                                                                                                                            
Disklabel type: gpt                                                                                                                                                                          
Disk identifier: 7B5E2580-54F7-4168-94B4-84B6A98C66F6                                                                                                                                        
                                                                                                                                                                                             
Device       Start      End  Sectors  Size Type                                                                                                                                              
/dev/vda1     2048     4095     2048    1M BIOS boot                                                                                                                                         
/dev/vda2     4096   264191   260096  127M EFI System                                                                                                                                        
/dev/vda3   264192  1050623   786432  384M Linux filesystem                                                                                                                                  
/dev/vda4  1050624 20971486 19920863  9.5G Linux filesystem

but after I layered it on top here is what it shows:

[core@cosa-devsh ~]$ sudo parted -l /dev/vda 
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name        Flags
 1      1049kB  2097kB  1049kB               BIOS-BOOT   bios_grub
 2      2097kB  135MB   133MB   fat16        EFI-SYSTEM  boot, esp
 3      135MB   538MB   403MB   ext4         boot
 4      538MB   10.7GB  10.2GB  xfs          root

Maybe it's because it's fat16? Some context on that in #993 (comment)

@dustymabe
Copy link
Member

Maybe a dumb question; but why can't FCOS add udisks2? There is a real use case here for mounting a partition dynamically. Isn't the space trade off worth it?

That's something that would have to be evaluated.

@hughsie
Copy link

hughsie commented Dec 7, 2023

Maybe it's because it's fat16

Maybe! What's the "Type" on the UDisk Partition interface please? I did think it might be ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 i.e. a BDP.

@dustymabe
Copy link
Member

Here's as much info that lsblk will output about the disk:

{
   "blockdevices": [
      {
         "alignment": 0,
         "id-link": "virtio-primary-disk",
         "id": "primary-disk",
         "disc-aln": 0,
         "dax": false,
         "disc-gran": "512B",
         "disk-seq": 1,
         "disc-max": "2G",
         "disc-zero": false,
         "fsavail": null,
         "fsroots": [
             null
         ],
         "fssize": null,
         "fstype": null,
         "fsused": null,
         "fsuse%": null,
         "fsver": null,
         "group": "disk",
         "hctl": null,
         "hotplug": false,
         "kname": "vda",
         "label": null,
         "log-sec": 512,
         "maj:min": "252:0",
         "min-io": 512,
         "mode": "brw-rw----",
         "model": null,
         "mq": " 12",
         "name": "vda",
         "opt-io": 0,
         "owner": "root",
         "partflags": null,
         "partlabel": null,
         "partn": null,
         "parttype": null,
         "parttypename": null,
         "partuuid": null,
         "path": "/dev/vda",
         "phy-sec": 512,
         "pkname": null,
         "pttype": "gpt",
         "ptuuid": "ed4b2e9c-541b-4046-a9ee-25c8c6d35d3c",
         "ra": 128,
         "rand": false,
         "rev": null,
         "rm": false,
         "ro": false,
         "rota": true,
         "rq-size": 256,
         "sched": "none",
         "serial": "primary-disk",
         "size": "10G",
         "start": null,
         "state": null,
         "subsystems": "block:virtio:pci",
         "mountpoint": null,
         "mountpoints": [
             null
         ],
         "tran": "virtio",
         "type": "disk",
         "uuid": null,
         "vendor": "0x1af4",
         "wsame": "0B",
         "wwn": null,
         "zoned": "none",
         "zone-sz": "0B",
         "zone-wgran": "0B",
         "zone-app": "0B",
         "zone-nr": 0,
         "zone-omax": 0,
         "zone-amax": 0,
         "children": [
            {
               "alignment": 0,
               "id-link": "virtio-primary-disk-part1",
               "id": "primary-disk-part1",
               "disc-aln": 0,
               "dax": false,
               "disc-gran": "512B",
               "disk-seq": 1,
               "disc-max": "2G",
               "disc-zero": false,
               "fsavail": null,
               "fsroots": [
                   null
               ],
               "fssize": null,
               "fstype": null,
               "fsused": null,
               "fsuse%": null,
               "fsver": null,
               "group": "disk",
               "hctl": null,
               "hotplug": false,
               "kname": "vda1",
               "label": null,
               "log-sec": 512,
               "maj:min": "252:1",
               "min-io": 512,
               "mode": "brw-rw----",
               "model": null,
               "mq": " 12",
               "name": "vda1",
               "opt-io": 0,
               "owner": "root",
               "partflags": null,
               "partlabel": "BIOS-BOOT",
               "partn": 1,
               "parttype": "21686148-6449-6e6f-744e-656564454649",
               "parttypename": "BIOS boot",
               "partuuid": "83dc181c-2be4-4375-9901-26b8a6fa9616",
               "path": "/dev/vda1",
               "phy-sec": 512,
               "pkname": "vda",
               "pttype": "gpt",
               "ptuuid": "ed4b2e9c-541b-4046-a9ee-25c8c6d35d3c",
               "ra": 128,
               "rand": false,
               "rev": null,
               "rm": false,
               "ro": false,
               "rota": true,
               "rq-size": 256,
               "sched": "none",
               "serial": null,
               "size": "1M",
               "start": 2048,
               "state": null,
               "subsystems": "block:virtio:pci",
               "mountpoint": null,
               "mountpoints": [
                   null
               ],
               "tran": "virtio",
               "type": "part",
               "uuid": null,
               "vendor": null,
               "wsame": "0B",
               "wwn": null,
               "zoned": "none",
               "zone-sz": "0B",
               "zone-wgran": "0B",
               "zone-app": "0B",
               "zone-nr": 0,
               "zone-omax": 0,
               "zone-amax": 0
            },{
               "alignment": 0,
               "id-link": "virtio-primary-disk-part2",
               "id": "primary-disk-part2",
               "disc-aln": 0,
               "dax": false,
               "disc-gran": "512B",
               "disk-seq": 1,
               "disc-max": "2G",
               "disc-zero": false,
               "fsavail": null,
               "fsroots": [
                   null
               ],
               "fssize": null,
               "fstype": "vfat",
               "fsused": null,
               "fsuse%": null,
               "fsver": "FAT16",
               "group": "disk",
               "hctl": null,
               "hotplug": false,
               "kname": "vda2",
               "label": "EFI-SYSTEM",
               "log-sec": 512,
               "maj:min": "252:2",
               "min-io": 512,
               "mode": "brw-rw----",
               "model": null,
               "mq": " 12",
               "name": "vda2",
               "opt-io": 0,
               "owner": "root",
               "partflags": null,
               "partlabel": "EFI-SYSTEM",
               "partn": 2,
               "parttype": "c12a7328-f81f-11d2-ba4b-00a0c93ec93b",
               "parttypename": "EFI System",
               "partuuid": "a43d1cd3-4b93-4b95-8bc3-cd80c6655aed",
               "path": "/dev/vda2",
               "phy-sec": 512,
               "pkname": "vda",
               "pttype": "gpt",
               "ptuuid": "ed4b2e9c-541b-4046-a9ee-25c8c6d35d3c",
               "ra": 128,
               "rand": false,
               "rev": null,
               "rm": false,
               "ro": false,
               "rota": true,
               "rq-size": 256,
               "sched": "none",
               "serial": null,
               "size": "127M",
               "start": 4096,
               "state": null,
               "subsystems": "block:virtio:pci",
               "mountpoint": null,
               "mountpoints": [
                   null
               ],
               "tran": "virtio",
               "type": "part",
               "uuid": "6C29-D5DD",
               "vendor": null,
               "wsame": "0B",
               "wwn": null,
               "zoned": "none",
               "zone-sz": "0B",
               "zone-wgran": "0B",
               "zone-app": "0B",
               "zone-nr": 0,
               "zone-omax": 0,
               "zone-amax": 0
            },{
               "alignment": 0,
               "id-link": "virtio-primary-disk-part3",
               "id": "primary-disk-part3",
               "disc-aln": 0,
               "dax": false,
               "disc-gran": "512B",
               "disk-seq": 1,
               "disc-max": "2G",
               "disc-zero": false,
               "fsavail": "219.8M",
               "fsroots": [
                   "/"
               ],
               "fssize": "349.9M",
               "fstype": "ext4",
               "fsused": "106.9M",
               "fsuse%": "31%",
               "fsver": "1.0",
               "group": "disk",
               "hctl": null,
               "hotplug": false,
               "kname": "vda3",
               "label": "boot",
               "log-sec": 512,
               "maj:min": "252:3",
               "min-io": 512,
               "mode": "brw-rw----",
               "model": null,
               "mq": " 12",
               "name": "vda3",
               "opt-io": 0,
               "owner": "root",
               "partflags": null,
               "partlabel": "boot",
               "partn": 3,
               "parttype": "0fc63daf-8483-4772-8e79-3d69d8477de4",
               "parttypename": "Linux filesystem",
               "partuuid": "f41424b8-0be1-4af3-a0ae-21c3e62eaf92",
               "path": "/dev/vda3",
               "phy-sec": 512,
               "pkname": "vda",
               "pttype": "gpt",
               "ptuuid": "ed4b2e9c-541b-4046-a9ee-25c8c6d35d3c",
               "ra": 128,
               "rand": false,
               "rev": null,
               "rm": false,
               "ro": false,
               "rota": true,
               "rq-size": 256,
               "sched": "none",
               "serial": null,
               "size": "384M",
               "start": 264192,
               "state": null,
               "subsystems": "block:virtio:pci",
               "mountpoint": "/boot",
               "mountpoints": [
                   "/boot"
               ],
               "tran": "virtio",
               "type": "part",
               "uuid": "5e6d7683-f34d-4dd5-9ecf-d31a9aa3fa7e",
               "vendor": null,
               "wsame": "0B",
               "wwn": null,
               "zoned": "none",
               "zone-sz": "0B",
               "zone-wgran": "0B",
               "zone-app": "0B",
               "zone-nr": 0,
               "zone-omax": 0,
               "zone-amax": 0
            },{
               "alignment": 0,
               "id-link": "virtio-primary-disk-part4",
               "id": "primary-disk-part4",
               "disc-aln": 0,
               "dax": false,
               "disc-gran": "512B",
               "disk-seq": 1,
               "disc-max": "2G",
               "disc-zero": false,
               "fsavail": "7.9G",
               "fsroots": [
                   "/ostree/deploy/fedora-coreos/var", "/ostree/deploy/fedora-coreos/var", "/ostree/deploy/fedora-coreos/deploy/e47320feb0004175a36f45ed361c4030af47ac7c148a5388340e74ce5ec86451.0/usr", "/ostree/deploy/fedora-coreos/deploy/e47320feb0004175a36f45ed361c4030af47ac7c148a5388340e74ce5ec86451.0/etc", "/ostree/deploy/fedora-coreos/deploy/e47320feb0004175a36f45ed361c4030af47ac7c148a5388340e74ce5ec86451.0", "/"
               ],
               "fssize": "9.4G",
               "fstype": "xfs",
               "fsused": "1.5G",
               "fsuse%": "16%",
               "fsver": null,
               "group": "disk",
               "hctl": null,
               "hotplug": false,
               "kname": "vda4",
               "label": "root",
               "log-sec": 512,
               "maj:min": "252:4",
               "min-io": 512,
               "mode": "brw-rw----",
               "model": null,
               "mq": " 12",
               "name": "vda4",
               "opt-io": 0,
               "owner": "root",
               "partflags": null,
               "partlabel": "root",
               "partn": 4,
               "parttype": "0fc63daf-8483-4772-8e79-3d69d8477de4",
               "parttypename": "Linux filesystem",
               "partuuid": "7e3fa89f-46ab-48a4-94a8-14e4d84269e9",
               "path": "/dev/vda4",
               "phy-sec": 512,
               "pkname": "vda",
               "pttype": "gpt",
               "ptuuid": "ed4b2e9c-541b-4046-a9ee-25c8c6d35d3c",
               "ra": 128,
               "rand": false,
               "rev": null,
               "rm": false,
               "ro": false,
               "rota": true,
               "rq-size": 256,
               "sched": "none",
               "serial": null,
               "size": "9.5G",
               "start": 1050624,
               "state": null,
               "subsystems": "block:virtio:pci",
               "mountpoint": "/sysroot",
               "mountpoints": [
                   "/var", "/sysroot/ostree/deploy/fedora-coreos/var", "/usr", "/etc", "/", "/sysroot"
               ],
               "tran": "virtio",
               "type": "part",
               "uuid": "0dab8650-ada1-4b2e-b281-3a6bc370f228",
               "vendor": null,
               "wsame": "0B",
               "wwn": null,
               "zoned": "none",
               "zone-sz": "0B",
               "zone-wgran": "0B",
               "zone-app": "0B",
               "zone-nr": 0,
               "zone-omax": 0,
               "zone-amax": 0
            }
         ]
      }
   ]
}

The EFI partition is set to c12a7328-f81f-11d2-ba4b-00a0c93ec93b (set here), which matches EFI System partition from the table in https://en.wikipedia.org/wiki/GUID_Partition_Table, which is correct IIUC.

@hughsie
Copy link

hughsie commented Dec 7, 2023

The EFI partition is set to c12a7328-f81f-11d2-ba4b-00a0c93ec93b

Ya, that's what we're looking for. I think the issue now is that we don't have the mechanism in place to mount and unmount the ESP.

@dustymabe
Copy link
Member

The EFI partition is set to c12a7328-f81f-11d2-ba4b-00a0c93ec93b

Ya, that's what we're looking for. I think the issue now is that we don't have the mechanism in place to mount and unmount the ESP.

But in the second half of #1623 (comment) I had it already mounted and EspLocation=/boot/efi set, so it shouldn't have needed any action?

@jlebon
Copy link
Member

jlebon commented Dec 7, 2023

So how is fwupd supposed to mount the partition on demand? EspLocation=/boot/efi is only going to work if the partition is already mounted. I don't mind using something other than udisks for FCOS, but shelling out to /usr/bin/mount is pretty janky at best...

Yeah, adding udisks2 can certainly be discussed. One alternative I think is to have fwupd have a built-in fallback where it does a simpler scan for ESPs, which I understand might not be desirable. Has something like that come up in fwupd already?

Another option is enhancing fwupd.conf to add a new e.g. EspDevices=/dev/foo,/dev/bar or something and FCOS shipping code to set those appropriately, which isn't ideal either.

@hughsie
Copy link

hughsie commented Dec 7, 2023

where it does a simpler scan for ESPs

Using libparted? I'm not hugely enamoured about adding additional library deps -- and I'm wondering if that package is in FCOS already either! :)

@superm1
Copy link

superm1 commented Dec 7, 2023

Yeah, adding udisks2 can certainly be discussed. One alternative I think is to have fwupd have a built-in fallback where it does a simpler scan for ESPs, which I understand might not be desirable. Has something like that come up in fwupd already?

Using libparted? I'm not hugely enamoured about adding additional library deps -- and I'm wondering if that package is in FCOS already either! :)

Furthermore we went through a lot of effort recently to remove extra libraries from fwupd to drop attack surface. It's not the right direction for us to add them back in.

Another option is enhancing fwupd.conf to add a new e.g. EspDevices=/dev/foo,/dev/bar or something and FCOS shipping code to set those appropriately, which isn't ideal either.

The problem with this is that fwupd still can't mount it if you told it a character device. It uses udisks to do the detection and mounting.

A 3rd option is to teach bootctl how to share it's discovery information with other software. It has it's own ESP detection algorithms AFAIK. But again we need the ESP to be mounted before the daemon starts if udisks isn't there because fwupd doesn't mount anything natively.

@jlebon
Copy link
Member

jlebon commented Dec 7, 2023

A 3rd option is to teach bootctl how to share it's discovery information with other software. It has it's own ESP detection algorithms AFAIK.

Yes, that's an interesting idea. That would ensure bootupd and fwupd are consistent in how the ESPs are found.

But again we need the ESP to be mounted before the daemon starts if udisks isn't there because fwupd doesn't mount anything natively.

OK, I didn't realize udisks2 also took care of the mounting part there. One thing to note though is that we don't want to mount the EFI partition in the global namespace. Hmm, looks like both fwupd.service and bootupd.service already use PrivateTmp=yes. So we could add e.g. JoinsNamespaceOf=fwupd.service in the latter and the bootupd command could mount the EFI partitions in e.g. /var/tmp/efiN.

@superm1
Copy link

superm1 commented Dec 7, 2023

We only want one ESP mounted. I like this though, no changes to fwupd. The conffile would just specify /var/tmp/efi or so as ESP location.

@cgwalters
Copy link
Member

One thing of note is that eventually I'd still like to do coreos/bootupd#132 - in which case, bootupd really needs to own mounting the ESP as there could be multiple, and we'd want to ensure synchronization.

So we could perhaps add bootupd mount-esp /some/target/path or so.

As far as mounting it today, by far the simplest is to assume the ESP is co-located with the root device; situations where it's not seem like they'd be really rare. So scanning the root block device for filesystems with part UUID C12A7328-F81F-11D2-BA4B-00A0C93EC93B would be quite simple.

@jlebon
Copy link
Member

jlebon commented Dec 8, 2023

Hmm, looks like both fwupd.service and bootupd.service already use PrivateTmp=yes. So we could add e.g. JoinsNamespaceOf=fwupd.service in the latter and the bootupd command could mount the EFI partitions in e.g. /var/tmp/efiN.

OK, this doesn't quite work actually. It looks like even with JoinsNamespaceOf=, mounts under there are not visible to the other. I think because systemd sets the /var/tmp mount propagation to private.

One thing of note is that eventually I'd still like to do coreos/bootupd#132 - in which case, bootupd really needs to own mounting the ESP as there could be multiple, and we'd want to ensure synchronization.

I think the term synchronization here can be misleading so I'll clear some things up just in case. Note that the ESP partitions in the CoreOS RAID1 setup are not RAIDed. See coreos/butane#178 and coreos/fedora-coreos-config#794. But yes, they do need to be "manually synchronized" by ensuring that whatever we do to one is done to the other (coreos/bootupd#132 is about teaching that to bootupd).

We could have bootupd abstract over this (I found coreos/bootupd#1 (comment) which suggested this) if we don't want to teach it to fwupd too. But OTOH, I would expect fwupd to also be interested in understanding this setup natively.

We only want one ESP mounted.

I think we can set it aside for now, but I agree with Colin, we really should address this eventually (for bootupd and also for fwupd). Otherwise, it's not well-defined how well fwupd works in such a RAID1 setup.

@jlebon
Copy link
Member

jlebon commented Dec 8, 2023

I think this might be worth a high-bandwidth meeting to discuss the path forward.

@superm1
Copy link

superm1 commented Dec 11, 2023

I think this might be worth a high-bandwidth meeting to discuss the path forward.

FYI - I'm not against a fwupd change that uses whatever you guys decide (if it's not adding udisks2 to your image). Feel free to open a PR upstream with the outcome.

hughsie added a commit to fwupd/fwupd that referenced this issue Dec 11, 2023
@travier
Copy link
Member Author

travier commented Dec 12, 2023

What adding udisks2 would mean for FCOS:

$ sudo rpm-ostree install --dry-run udisks2
Checking out tree cd3ab59... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2023-03-14T10:57:01Z solvables: 4
rpm-md repo 'updates' (cached); generated: 2023-12-12T01:47:32Z solvables: 15785
rpm-md repo 'fedora' (cached); generated: 2023-11-01T00:12:39Z solvables: 70825
rpm-md repo 'updates-archive' (cached); generated: 2023-12-12T02:37:53Z solvables: 16502
Resolving dependencies... done
Installing 36 packages:
  crypto-policies-scripts-20231113-1.gitb402e82.fc39.noarch (updates)
  exfatprogs-1.2.2-1.fc39.x86_64 (updates)
  f2fs-tools-1.16.0-2.fc39.x86_64 (fedora)
  libatasmart-0.19-26.fc39.x86_64 (fedora)
  libb2-0.98.1-9.fc39.x86_64 (fedora)
  libblockdev-3.0.4-1.fc39.x86_64 (fedora)
  libblockdev-crypto-3.0.4-1.fc39.x86_64 (fedora)
  libblockdev-fs-3.0.4-1.fc39.x86_64 (fedora)
  libblockdev-loop-3.0.4-1.fc39.x86_64 (fedora)
  libblockdev-mdraid-3.0.4-1.fc39.x86_64 (fedora)
  libblockdev-nvme-3.0.4-1.fc39.x86_64 (fedora)
  libblockdev-part-3.0.4-1.fc39.x86_64 (fedora)
  libblockdev-swap-3.0.4-1.fc39.x86_64 (fedora)
  libblockdev-utils-3.0.4-1.fc39.x86_64 (fedora)
  libbytesize-2.10-1.fc39.x86_64 (updates)
  libgomp-13.2.1-6.fc39.x86_64 (updates)
  libudisks2-2.10.1-1.fc39.x86_64 (fedora)
  mpdecimal-2.5.1-7.fc39.x86_64 (fedora)
  nilfs-utils-2.2.9-4.fc39.x86_64 (fedora)
  nspr-4.35.0-14.fc39.x86_64 (fedora)
  nss-3.94.0-2.fc39.x86_64 (fedora)
  nss-softokn-3.94.0-2.fc39.x86_64 (fedora)
  nss-softokn-freebl-3.94.0-2.fc39.x86_64 (fedora)
  nss-sysinit-3.94.0-2.fc39.x86_64 (fedora)
  nss-util-3.94.0-2.fc39.x86_64 (fedora)
  ntfs-3g-2:2022.10.3-3.fc39.x86_64 (fedora)
  ntfs-3g-libs-2:2022.10.3-3.fc39.x86_64 (fedora)
  ntfs-3g-system-compression-1.0-14.fc39.x86_64 (fedora)
  ntfsprogs-2:2022.10.3-3.fc39.x86_64 (fedora)
  python-pip-wheel-23.2.1-1.fc39.noarch (fedora)
  python-unversioned-command-3.12.0-1.fc39.noarch (fedora)
  python3-3.12.0-1.fc39.x86_64 (fedora)
  python3-libs-3.12.0-1.fc39.x86_64 (fedora)
  udftools-2.3-7.fc39.x86_64 (fedora)
  udisks2-2.10.1-1.fc39.x86_64 (fedora)
  volume_key-libs-0.3.12-20.fc39.x86_64 (fedora)
Exiting because of '--dry-run' option

Edit: A lot of those are recommends that would not be included. Will do a full build to check.

@travier
Copy link
Member Author

travier commented Dec 12, 2023

Well, this pulls-in Python:

error: Could not depsolve transaction; 1 problem detected:
 Problem: package libblockdev-crypto-3.0.4-1.fc39.x86_64 from fedora requires libnss3.so()(64bit), but none of the providers can be installed
  - package libblockdev-crypto-3.0.4-1.fc39.x86_64 from fedora requires libnss3.so(NSS_3.2)(64bit), but none of the providers can be installed
  - package libblockdev-crypto-3.0.4-1.fc39.x86_64 from fedora requires libsmime3.so()(64bit), but none of the providers can be installed
  - package libblockdev-crypto-3.0.4-1.fc39.x86_64 from fedora requires libssl3.so()(64bit), but none of the providers can be installed
  - package libblockdev-crypto-3.0.4-1.fc39.x86_64 from fedora requires libnss3.so(NSS_3.9.2)(64bit), but none of the providers can be installed
  - package libblockdev-crypto-3.0.4-1.fc39.x86_64 from fedora requires libsmime3.so(NSS_3.4)(64bit), but none of the providers can be installed
  - package udisks2-2.10.1-1.fc39.x86_64 from fedora requires libblockdev-crypto >= 3.0, but none of the providers can be installed
  - package nss-3.94.0-2.fc39.x86_64 from fedora requires /usr/bin/update-crypto-policies, but none of the providers can be installed
  - package crypto-policies-scripts-20230731-1.git5ed06e0.fc39.noarch from fedora requires /usr/bin/python3, but none of the providers can be installed
  - package crypto-policies-scripts-20231113-1.gitb402e82.fc39.noarch from fedora-updates requires /usr/bin/python3, but none of the providers can be installed
  - package libblockdev-crypto-3.0.4-1.fc39.i686 from fedora requires libudev.so.1, but none of the providers can be installed
  - conflicting requests
  - package crypto-policies-scripts-20220815-1.gite4ed860.fc37.noarch from fedora-coreos-pool is filtered out by exclude filtering
  - package crypto-policies-scripts-20221215-2.gita4c31a3.fc38.noarch from fedora-coreos-pool is filtered out by exclude filtering
  - package crypto-policies-scripts-20230301-1.gita12f7b2.fc38.noarch from fedora-coreos-pool is filtered out by exclude filtering
  - package crypto-policies-scripts-20230731-1.git5ed06e0.fc39.noarch from fedora-coreos-pool is filtered out by exclude filtering
  - package crypto-policies-scripts-20231113-1.gitb402e82.fc39.noarch from fedora-coreos-pool is filtered out by exclude filtering
  - package python3-3.12.0-1.fc39.i686 from fedora is filtered out by exclude filtering
  - package python3-3.12.0-1.fc39.x86_64 from fedora is filtered out by exclude filtering
  - package systemd-libs-254.5-2.fc39.i686 from fedora is filtered out by exclude filtering
  - package systemd-libs-254.7-1.fc39.i686 from fedora-updates is filtered out by exclude filtering

udisks2 -> libblockdev-crypto -> nss -> crypto-policies-scripts -> /usr/bin/python3

@travier
Copy link
Member Author

travier commented May 15, 2024

This is udisks2 -> libblockdev-crypto -> nss -> crypto-policies-scripts -> python3 so this is not just about Python but also adding another cryptographic stack (nss) to Fedora CoreOS on top of what we already have (openssl).

@hughsie
Copy link

hughsie commented May 15, 2024

So assuming we could hardcode the ESP location in FCOS -- how would fwupd mount it? system("mount /dev/foo") is pretty terrible. How does FCOS deal with disks right now?

@superm1
Copy link

superm1 commented May 15, 2024

This is udisks2 -> libblockdev-crypto -> nss -> crypto-policies-scripts -> python3 so this is not just about Python but also adding another cryptographic stack (nss) to Fedora CoreOS on top of what we already have (openssl).

But isn't this a packaging problem? Why do a bunch of scripts get pulled in?

@travier
Copy link
Member Author

travier commented May 16, 2024

how would fwupd mount it? system("mount /dev/foo") is pretty terrible. How does FCOS deal with disks right now?

Mount points are managed by systemd mount units for all normal partitions. We don't have a /etc/fstab by default.

In previous changes, we removed the mount unit for the ESP:

So ideally, fwupd should unshare the mount namespace and then mount the EFI partition to /boot/efi.

How it's done in bootupd:

We are still missing support for the RAID case however: coreos/bootupd#132

@hughsie
Copy link

hughsie commented May 16, 2024

fwupd should unshare the mount namespace and then mount the EFI partition to /boot/efi

Can we ask bootupd to do that for us? I thought that was the thing controlling access to the ESP now?

@travier
Copy link
Member Author

travier commented May 16, 2024

It's not really controlling access to the ESP but it's the only one besides fwupd that should need access to it. We could extend bootupd to allow fwupd to run a command in a namespace with the ESP mounted for example so that fwupd does not have to "learn how to do it".

@superm1
Copy link

superm1 commented May 16, 2024

fwupd expects that the daemon is mountable/umountable at runtime, not so much a "command". Can we have a dbus interface to let fwupd mount/umount?

@hughsie
Copy link

hughsie commented May 16, 2024

Well, kinda -- we could say "write this file descriptor to $ESP/fedora/fw/foo.cap` for example.

@cgwalters
Copy link
Member

Maybe how it should work is that bootupd just offers an API to mount to a specified empty dir, and in the future when we do RAID we'd make it a FUSE mount or so, and synchronize the changes. Or it could be one of the real underlying dirs, but we e.g. always bump the root timestamp, and then know we need to synchronize other copies to the one with the newest timestamp.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants