Change EFI-System partition format from fat16 to fat32

[buckaroogeek]

Bug

Raspberry Pi 4 will not boot Fedora CoreOS if the EFI-System partition is FAT16. Convert to FAT32 and the RPi4 boots. The coreos-installer appears to use FAT16 for the EFI system partition. I could not see an option to change this at coreos-installer run time.

Host Operating System Version

Fedora 34

Target Operating System Version

Fedora CoreOS 34

coreos-installer Version

coreos-installer-0.10.0-2.fc34.x86_64

Expected Behavior

Use coreos-installer to create a bootable USB mass storage device for Raspberry Pi 4. CoreOS should boot once coreos-installer completes without error and the USB device is plugged into the RPi4 and powered on.

Actual Behavior

FCOS will not boot on the RPi4 until the EFI-System partition is reformatted as FAT32. It appears that coreos-installer formats the partition as FAT16.
The RPi4 displays a message: Trying partition 0. Trying to read partition as fat. (with other text), This message block repeats blocking the boot process.

Reproduction Steps

1. Install FCOS to usb (e.g. sudo coreos-installer install /dev/sdh -i ./fc1.ign -f fedora-coreos-34.20210919.3.0-etal.aarch64.raw --offline --insecure)
2. Dismount USB from Fedora workstation
3. Mount USB on RPi4 (usb3 port).
4. Power on.
5. RPi4 cannot read partition.

Other Information

I am using v 1.31 of the pftp/uefi firmware. I can manually reformat the partition to FAT32 and FCOS boots as expected.

[jlebon]

As far as I can tell from the UEFI specification, the firmware is supposed to support FAT16 or FAT32:

13.3.1.1 File System Format

The EFI firmware must support the FAT32, FAT16, and FAT12 variants of the EFI file system. What variant
of EFI FAT to use is defined by the size of the media. The rules defining the relationship between media
size and FAT variants is defined in the specification for the EFI file system.

We seem to not be specifying the actual FAT size when running mkfs.fat, which looks like it makes it also auto-detect the right one based on size:

-F FAT-SIZE
           Specifies  the type of file allocation tables used (12, 16 or 32 bit).  If nothing is speci‐
           fied, mkfs.fat will automatically select between 12, 16 and 32 bit, whatever fits better for
           the filesystem size.

which seems to fit with the UEFI definition.

AFAIK, we've had other users successfully boot FCOS on the Raspberry Pi 4. I suspect this may be related to the fact that you're booting via USB instead of SD card? @dustymabe Have you tried booting via USB?

[bgilbert]

My understanding is that the ESP on a non-removable disk must be FAT32. UEFI spec 2.9 section 13.3 says:

EFI encompasses the use of FAT32 for a system partition, and FAT12 or FAT16 for removable media.

We had the same bug in Container Linux (coreos/bugs#2246).

[buckaroogeek]

Thanks for the information about the specification and updates. I should spell check before I submit! I recall that during my initial efforts to install and run FCOS on RPi4 hardware last February (see https://forums.raspberrypi.com/viewtopic.php?t=304318 for a write up if interested), this was not an issue I ran into. The USB disk I uses is a KINGSTON SA400S37120G (120 GB). Please let me know if I can provide any more information and/or testing.

[dustymabe]

@dustymabe Have you tried booting via USB?

I've been recently testing things via both uboot and UEFI (working on documentation to add to our docs site). I have to admit I have seen some cases where the pi4 will loop on trying partition 0 like @buckaroogeek suggested, but I've also taken that exact USB out and plugged it back in and it worked the next time, which was odd.

Maybe use of FAT16 makes the issue more likely to happen? Maybe I'm misremembering? I definitely know things are working on an SD card just fine the way it is today.

[buckaroogeek]

I checked the eeprom firmware page on github (https://github.com/raspberrypi/rpi-eeprom/blob/master/firmware/release-notes.md) and see this note: "Update the FAT detection to support FAT16 for EFI/ESD paritions with GPT instead of assuming FAT32. The latest firmware is also required for a similar update." This is for a 4 Oct update.

[bgilbert]

This is a corner case that likely affects more than just the Pi, so we should fix it regardless. Fix in coreos/coreos-assembler#2491.

[bgilbert]

And thanks for the report!

[buckaroogeek]

Glad to help.

[dustymabe]

Going to re-open until the change lands

[bgilbert]

There's a complication. Microsoft's EFI FAT32 specification says:

The FAT type—one of FAT12, FAT16, or FAT32—is determined by the count of clusters on the volume and nothing else.

FAT32 is defined to have at least 65525 clusters, and with one 4 KiB sector/cluster, that means the ESP on a 4Kn disk needs to be at least 257 MiB. Otherwise EDK II (and presumably other UEFI firmware) won't accept it as a valid filesystem. Our ESP is 127 MiB, and the partition layout is difficult to change because it's also wired into Butane.

We could switch to FAT32 only in the 512b image, which would improve the situation somewhat, but would create a weird inconsistency between our image types. Or we could change the partition layout again, or stick with FAT16.

[bgilbert]

If we switched the ESP to FAT32, we should probably update the Butane RAID template to do the same. Butane doesn't have any way to know the boot disk's sector size (the user may not know either), so effectively it can't use different templates for 512b and 4Kn disks.

[dustymabe]

discuss at meeting?

[bgilbert]

Yeah. Added meeting label.

[jlebon]

Was curious about what Anaconda did here. It uses blivet, which looks like it currently always creates a FAT16 filesystem: https://github.com/storaged-project/blivet/blob/314675891c86ba185deabc52f77ca6389973d9a1/blivet/formats/fs.py#L925-L926

[lucab]

  * AGREED: For now, we will document workarounds for systems that can't
    boot from a FAT16 ESP, including older Raspberry Pi firmware.  Next
    time we change the partition layout, we will switch the ESP to FAT32
    in both the 512b and 4Kn images, selecting a size of at least 257 MB
    to fit a valid FAT32 filesystem.

[buckaroogeek]

My thanks for the thorough review of this issue. Documentation as noted by @lucab will be helpful and useful. Shall I close or wait for the documentation?

[bgilbert]

I've filed coreos/fedora-coreos-docs#326 to track the docs issue. Let's leave this open, though, since we'd still like to switch to FAT32 in the future.

For the record, there's a lot of good discussion in the meeting logs. pjones pointed out that we're not violating the UEFI spec here because the spec isn't binding on us; we're just depending on firmware behavior which is apparently not mandated by the spec.

[bgilbert]

@dustymabe pointed out that enlarging the ESP implicates #586. We addressed that issue by recommending a minimum root filesystem size, which doesn't help if the same-size rootfs is placed after a larger ESP.

[dustymabe]

For at least the RPi4 part of this I added a section to the docs in coreos/fedora-coreos-docs#336 that talks about how to update the EEPROM. The docs URL is: https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-raspberry-pi4/#_updating_eeprom_on_raspberry_pi_4

[cgwalters]

This intersects with the UKI bits - with UKIs, we don't need a separate /boot partition, so we could use that space at least to expand the ESP.