POC: Dynamically load ignition and afterburn from rootfs

[cgwalters]

This is a proof-of-concept of the idea in
coreos/fedora-coreos-tracker#1247 (comment)

The role of the initramfs originally was just to mount the root filesystem.
Us running ignition from the initramfs makes sense, but it doesn't mean
the ignition binary has to physically live in the initramfs.

In the end state our initramfs for example doesn't need to physically
contain NetworkManager for example either. Or for that matter, kernel network drivers.

It just has to have enough code to mount the root filesystem, and
neither ignition nor afterburn are needed for that.

This clearly adds some nontrivial logic to our already nontrivial
initramfs. But, it does shave 9M (after compression) from each copy of the initramfs,
so in the likely case of having (transiently) 3 different versions,
we will save 27MB in /boot, which is a good amount.

[cgwalters]

Yes, this hacky bit relies on the fact that we depend on the afterburn and dracut modules, which will have installed the binaries first, then we remove them.

[cgwalters]

We could shrink this to only special casing ignition, which would reduce some complexity in dealing with systemd unit dependencies (afterburn-in-initramfs is special in that it runs both before ignition and also after the real root switch).

Also, anything of this form will need special build logic, probably in coreos-assembler to regenerate a "full" initramfs for PXE cases.

[cgwalters]

Hmm. Or maybe what would actually be a lot simpler is to build (using dracut) two initramfs images (which we need anyways for PXE), one that e.g. omits ignition and afterburn and NetworkManager etc., and the second which includes them. Our build process could basically diff them, and auto-generate code to copy all the files from the second into the first after mounting the rootfs.

[cgwalters]

and auto-generate code to copy all the files from the second into the first after mounting the rootfs.

Though, the ugly part about that is we'd omit the systemd units...so we'd have to do something ugly like a systemctl daemon-reload in the middle of boot. Probably best to only omit ELF binaries which are the big things.

OK so something like:

dracut -f initramfs-full.img
dracut --omit "ignition networkmanager afterburn ..." initramfs-diskful.img
(mkdir full && cd full && lsinitrd --unpack initramfs-diskful.img)
(mkdir diskful && cd diskful && lsinitrd --unpack initramfs-diskful.img)
(cd full && find -type f) | while read file; do 
  if test '!' -f diskful/$file && file ${file} | grep -q ': ELF'; then
    rm diskful/$file
    echo $file >> diskful/etc/removed-manifest.txt 
  fi
done
dracut --initramfs-from-fs diskful initramfs-diskful.img  # regenerate it

Then we have code already in the initramfs which parses /etc/removed-manifest.txt or so and loads from the real root.

[cgwalters]

bgilbert had a notable point in the original issue that relates to this:

We can't further improve this by kicking out NetworkManager and network drivers, since we need those for Tang-bound disk encryption.

Ah, right. That does limit us here.