Root-embedded UKI

[Winterhuman]

https://0pointer.net/blog/brave-new-trusted-boot-world.html mentions under Whole OS embedded in the UKI that you could potentially embed a root FS into a UKI, is there any documentation for how to go about creating such a UKI?

[poettering]

You could use a tool like mkosi for that, see https://github.com/systemd/mkosi

But it's not a ready-made solution for it.

1. It can generate "cpio" archives for your from a distro, which is suitable as the initrd for such a UKI.
2. For regular OS images it is able to combined kernel+initrd into a UKI and invoke systemd-measure and sign it, and use the distro initrd builder for that.

but what is currently missing is to generate the "cpio" first, and then do the objcopy/systemd-measure/sbsign stuff with that so that the primary output is a UKI.

@behrmann @DaanDeMeyer what's your take on this?

[DaanDeMeyer]

Once I finish the refactor to use systemd-repart in mkosi and all the associated changes to the build process, it'll be trivial to add a new output format that generates a UKI with rootfs embedded. But I want to do all the refactoring stuff first before adding more stuff.

(Which means I need all the systemd-repart PRs in systemd merged, so I need reviews for those!)

[teg]

@gicmo has been looking into doing this for image builder. I think for now we were looking at embedding a squashfs in the initrd, but correct me if I'm wrong :)

[gicmo]

I indeed did and it was quite straight forward. The code currently only lives in my uki branch, but I am planning on filing a proper PR next week.

[Winterhuman]

@poettering Just as a thought, normal UKIs can be constructed manually with objcopy, is it possible to construct a objcopy command to embed a root FS into a UKI? Or is embedding a bit more involved than simply copying it in?

[poettering]

when systemd is invoked as PID 1 it checks for the existance of /etc/initrd-release. If it exists it assumes it runs in an initrd and then starts initrd.target (see bootup(7) man page) to ultimately transition into the real root. If it doesn't exist it assumes it runs in the real root and invokes default.target instead.

Hence, if you build an initrd cpio but leave /etc/initrd-release out you'll get what you are asking for: a real system that runs from the initrd cpio unpacked into a tmpfs, and there's never a transition into the real host.

[poettering]

(you can use the check for /etc/initrd-release actually the opposite way too: some environments have a trivial initrd built into the UKI, and then a secondary initrd externally in the ESP, and that transitions to the real root. In such a model you'd put /etc/initrd-release in both the primary and the secondary initrd, to mark them as such, and tell systemd about it. But the secondary initrd isn't really an initrd strictly speaking because the kernel didn't unpack it from a tmpfs, but the primary initrd already transitioned into it)

[Winterhuman]

Thanks @poettering! I wonder how possible it is to have one UKI be an initrd for another UKI, but keep the resources from the first UKI available somehow (perhaps using overlayfs to merge them?). Just thinking of ways you could go past the 4GiB size limit (or maybe just for partitioning specific resources into specific UKIs [which sounds like a very complicated way to reinvent partitions... hmm])

[poettering]

UKIs are primarily kernel images, which then also contain an initrd, hence not sure I can parse "have one UKI be an initrd for another UKI"?

Note that the initrd is in RAM during runtime. Having more than 4G pinned like that in RAM is probably not the greatest of ideas hence the size limit should not be much of an issue...

There are systems which boot an initrd embedded in a UKI as main root fs, and then just use systemd-sysext, systemd's Portable Services and systemd-nspawn containers for the actual payload.

[julian-klode]

Related, dinner thought today was wondering if the UKI could contain a squashfs (erofs) image instead in addition such that we could just keep the kernel bits in memory and then map the rest from the device. Not clear what needs to remain in memory once the kernel executes when the kernel image is loaded from an UKI.

Because let's say you want to have a single file root-embedded UKI as a deployment method and deploy that to like 128MB RAM systems - you do not want to keep the rootfs in memory when executing the user space.

[poettering]

@julian-klode you could include the erofs loopback file in the cpio, then set it up via /dev/loop* and then continue running. Which would mean it's not backed by storage. If you want that, two ideas:

1. activate a swap device. because the initrd cpio is backed by tmpfs these days, and tmpfs is swappable memory, this is sufficient.

2. if you can't spare the swap device, and want to use the copy in the UKI as backing storage, then that should work too: once you booted up, find the UKI in the ESP. Now use the LOOP_CHANGE_FD ioctl of the loopback device to magically swap out the backing storage on the fly from the cpio tmpfs to the ESP. There are two complications here: because the API sucks you cannot change the offset at the same time, but you really want this here, after all the erofs image is gonna be somewhere in the middle of the UKI, not at the beginning. But you can salvage that: create a new loopback device with the right offset, and then use LOOP_CHANGE_FD on the original loopback to this secondary loopback device (yes you can create loopback devices that are backed by loopback devices, and as long as you enable direct-io mode it shouldn't even cost you). the other issue is you need to ensure that the erofs is 4K aligned in the UKI. That requires two things: that the PE section is 4K aligned, which is trivial, objcopy allows you to do that. But then you also need to align the erofs in the cpio at a 4K boundary. Given that cpio knows now alignment concept you thus might need to insert an additionr regular file into the cpio stream that has no other purposes of inserting enough extra bytes into the stream so that the following erofs is at a 4K boundary.

[Winterhuman]

I've opened an issue for seeing if systemd could handle option 2, feel free to close it if it's a bit of a niche/complex use-case for systemd to support: systemd/systemd#25428

[arnout]

Buildroot can be used to create a rootfs that contains a set of selected packages. Several rootfs types are supported, one of them is the cpio archive typically used for an initrd. The initrd can even be linked into the kernel binary (though that is not necessary for an UKI image).
Buildroot currently doesn't have an option to generate a UKI image, but that would be a relatively simple extension.

[Winterhuman]

Just another idea to throw into the discussion, how feasible do you think it'd be to use swap space to persist state between OS reboots? So, something like:

1. A UKI, containing a root FS section, is booted by the firmware/bootloader.
2. systemd creates a loopback file for the root section, as discussed at Pivot to unloaded root sections in initramfs systemd/systemd#25428.
3. But then, systemd additionally uses something similar to systemd.volatile=overlay, but with support to hibernate the "volatile" layer allowing weak persistence between reboots (until swap is cleared).

Or moreover, can the tmpfs created through systemd.volatile=overlay be hibernated?