[RPI] EFI booting no longer supported on kernels shipped with ubuntu > 22.04

[Itxaka]

For some reason, the efi booting stuff was disabled on the upstream ubuntu kernel packages for raspi:

commit: https://git.launchpad.net/ubuntu/+source/linux-raspi/commit/?h=applied/ubuntu/devel&id=0d02ca7853f4e487ff8b912b0d1d43ceb31a528b

Change:

diff --git a/debian.raspi/config/annotations b/debian.raspi/config/annotations
index 5fddd3c1a9..136d8cf4af 100644

-CONFIG_EFI                                      policy<{'arm64': 'y', 'armhf': 'n'}> note<'Not supported on Pi'>
+CONFIG_EFI                                      policy<{'arm64': 'n', 'armhf': 'n'}> note<'Not supported on Pi'>

What this means is that for our usecase, we would not be able to boot with our current method uboot -> arm64 fallback file (grub) -> kernel as grub complains that the kernel doesnt have the required CONFIG_EFI_STUB option, which is true.

We should probably ask upstream why was this disabled and if they can enable it again, as it was enabled up to kernel 6.5.0-1001.1 with no apparent issues (could not find any bugs related to this)

[mudler]

one (hackish) way would be to use different kernels (even vanilla..) but this is not easy to maintain in the long run.

Probably upstream needs to be educated that actually works on PI (given their note "Not supported on Pi"), as u-boot > GRUB is quite uncommon.

However for now we can block RPI4 support for ubuntu >22.04 as currently there are no requests around that.

[Itxaka]

https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/2053147

[Itxaka]

bug was closed upstream as they wont provide support for it.

[mauromorales]

shoot, what does that mean, no RPI Ubuntu ? so strange since they offer an Ubuntu RPi release themselves

[Itxaka]

Yeah but our boot method is different I guess? @mudler probably know of the difference between the normal boot and our boot process, so maybe we need to change our boot process to be close to upstream for this.

[mauromorales]

yup good question, I think they don't use u-boot, but not sure

[mauromorales]

@kairos-io/maintainers should we stop producing Ubuntu RPi artifacts for 23+? IMO it is weird that we produce an artifact if we know it will not work, and have to keep adding a notice on the release

[Itxaka]

as last resort we should try https://github.com/pftf/RPi4 + systemd-boot with type 1 stuff to see if it works that way and its only grub the one being obtuse about it.

[Itxaka]

nah, I played wiht it, it would require waaaay to many changes to our current implementation of the boot process just for ubutu rpi :(

IMO, we stop producing rpi artifacts for ubuntu and point to the upstream bug. If people really want it they can go to the ticket and comment on it so maybe ubuntu would reconsider increasing the kernel size by 8Kb due to that config. (I can understand the increase in surface attack....mostly....the other 99% of kernels out there have it already enbabled but whatever...)

[liyimeng]

It is too bad that the upstream taking that decision. :(
However, people play with uboot probably have good knowledge with re-configuring kernel on their own. Maybe we could just update the docs, explaining how kairos is dealing with u-boot based modes, like Jetson, RPI, etc. It would be even better to add some instructions, helping developer to enable their board with KairOS, which will bring KairOS to much larger user base.

I know little in the area. However, I've tried to absorb some knowledge from your guys discussion and compose a list here. To have kairOS work with any boards, what is needed:

• Developer need to ship the uboot for their boards with UEFI support, detail instructions can be found at https://github.com/u-boot/u-boot/blob/master/doc/develop/uefi/uefi.rst
• Pack the firmware for the board with package manager tools, like rpm, deb, apk, etc.
• Build a custom kernel with necessary drivers and configs, especially CONFIG_EFI_STUB=y and pack it as above.

After this, KairOS can launch with uboot in EFI mode, which in turn runs grub.efi and then we load our grub entries from there.

Not sure if this makes sense. If yes, I can create a PR for the docs.

[liyimeng]

yaha, we actually already have a doc for this https://kairos.io/docs/development/nvidia/

[jimmykarily]

Idea that came up during planning: We can maybe create a ppa in launchpad (https://launchpad.net) to build the kernel ourselves and just use our package instead. This of course needs someone that knows how to build in launchpad...

[Itxaka]

Idea that came up during planning: We can maybe create a ppa in launchpad (launchpad.net) to build the kernel ourselves and just use our package instead. This of course needs someone that knows how to build in launchpad...

The actual code is simple, we just need a diff that changes 1 line

diff --git a/debian.raspi/config/annotations b/debian.raspi/config/annotations
index 5fddd3c1a9..136d8cf4af 100644

-CONFIG_EFI                                      policy<{'arm64': 'y', 'armhf': 'n'}> note<'Not supported on Pi'>
+CONFIG_EFI                                      policy<{'arm64': 'n', 'armhf': 'n'}> note<'Not supported on Pi'>

so ideally it would be

Fork with just that diff -> upstream commits a new versions -> sync fork -> apply patch -> build kernel in a PPA

But who knows how we can trigger that in launchpad... we could do manual checks in a custom repo but then we would not get a PPA.

We could also just use a different kernel for rpi. opensuse, archlinux, etc...

[Itxaka]

could also send a patch upstream, see if its accepted lol

[Itxaka]

or we could use the upstream linux kernel for rpi? https://github.com/raspberrypi/linux or even the raspberryos kernel?

[jimmykarily]

like this? https://git.launchpad.net/~dkarakasilis/ubuntu/+source/linux-raspi/commit/?id=917694f382899575eb587bb481cb02ef71d9821c

[Itxaka]

like this? git.launchpad.net/~dkarakasilis/ubuntu/+source/linux-raspi/commit?id=917694f382899575eb587bb481cb02ef71d9821c

sorcery!

[jimmykarily]

Uploading the artifact fails with:

DEBUG Considering changefile 3429361/ubuntu/linux-raspi_6.8.0-1~ubuntu24.04.1_source.changes
DEBUG Finding fresh policy
INFO Processing upload linux-raspi_6.8.0-1~ubuntu24.04.1_source.changes
INFO Upload was rejected:
INFO 	6.8.0-1~ubuntu24.04.1: should be 6.8.0-1011.12 according to changes file.
INFO Committing the transaction and any mails associated with this upload.

Maybe relevant: https://help.launchpad.net/PPAQuickStart/FAQ#I_get_an_error_about_versions

[jimmykarily]

I asked for some guidance in the launchpad users mailing list: https://lists.launchpad.net/launchpad-users/

[jimmykarily]

In the meantime, we can just dpkg -i https://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/raspberrypi-kernel_1.20230405-1_arm64.deb and see if it works?

(repo with the kernel: https://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/)

[jimmykarily]

I'm trying it here: https://github.com/kairos-io/kairos/compare/2249-try-upstream-rpi-kernel?expand=1 but it turns out they've removed some packages too?

    ubuntu-22.04-rpi *failed* | E: Unable to locate package linux-modules-extra-raspi
    ubuntu-22.04-rpi *failed* | ERROR
    ubuntu-22.04-rpi *failed* |       The command
    ubuntu-22.04-rpi *failed* |           RUN apt-get update && apt-get install -y --no-install-recommends     linux-firmware-raspi     linux-modules-extra-raspi     && apt-get clean && rm -rf /var/lib/apt/lists/*

I tried to see if this is a mistake in our Dockerfile, maybe the file is renamed or something?

[jimmykarily]

Maybe this? https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/2048862

[jimmykarily]

This is what it boots into:

Not excatly what we had in mind :D.

[jimmykarily]

Hitting ESC shows errors like:

Failed to open device: 'sdcard' (cmd 371a0010 status 1fff0001)

[jimmykarily]

We tried things, it doesn't work. We have other flavors that work. Let's close it.