Add factory reset capability

[bgilbert]

Under the immutable infrastructure model, we encourage users to handle config changes by reprovisioning their systems, rather than using configuration management or manually modifying the running system. This is okay in virtualized setups or in the cloud, and is also okay on bare metal where PXE infrastructure or IPMI-assisted ISO boot is available. But on single-node bare metal setups without external infrastructure, manual reinstall is impractical. (See e.g. this discussion.)

For this use case, it might help to have a command which receives an Ignition config to use for reprovisioning, copies it into /boot, sets first-boot kargs, and reboots. Then the initramfs, before running the usual first-boot steps, would need to restore the system to a pristine state. That process might look a lot like #94, or might look more like clearing /var and resetting /etc from /usr/etc. If those are in non-standard locations, perhaps the user-mode command could add kargs which tell the initramfs where to find them.

Because we'd be deleting user customizations before rerunning Ignition, this approach wouldn't[*] get into unsupported territory of using Ignition for configuration management. It would slightly complicate OS maintenance, though. At present we can freely change the Ignition initrd glue in ways that are incompatible with old installs (e.g. by changing first-boot kargs) because old installs will never run Ignition again. Supporting factory reset of old nodes would presumably add some additional constraints.

[*] Presumably it'd be infeasible to delete every possible user customization, so that statement might be optimistic.

[bgilbert]

Discussed in the community meeting today.

• If we committed the Ignition config to the ostree repo, that could make this a bit easier.
• coreos-installer used to be in the initramfs, which technically allowed every system to reinstall itself outright (assuming network access). We could investigate running the new coreos-installer from the initramfs again, but it requires lsblk, udevadm, and gpg.
• We've discussed splitting the live PXE root squashfs out to a separate stage2 (Publish the initrd and rootfs images separately for PXE booting #390) which could be useful here: assuming network access, the existing installed kernel and initrd could boot themselves into a live PXE image and rerun coreos-installer from there. That doesn't let air-gapped systems reprovision without external assistance, though.
  • Assuming network access, we could also fetch the image before the reboot and stash it somewhere on disk. But if the install fails and overwrites the image in the process, you don't get a second chance.
  • To solve the air-gapped case, we could: 1) before reboot, assemble a live squashfs from the ostree we already have, 2) from the initramfs, mount the old root filesystem and copy the squashfs into RAM, 3) create a temporary partition at the end of the disk and copy the squashfs into it for safety, 4) run coreos-installer, and if it fails, recreate /boot and the safety partition, 5) reboot into the new system. That's complex though.
• We didn't explore it in the meeting, but rerunning coreos-installer is not mandatory if there's a good way to delete user customizations from the existing system. We probably won't be able to get the system to 100% pristine state though.

There was a general sense that the feature would be nice, but there was also concern about implementation complexity depending on the approach taken.

[jlebon]

Note with the stage 2 approach, as long as the stage 2 binary is the initrd we append to the base one, it should contain both the osmet file and the root squashfs, so it should be possible to do the re-install completely offline.

[jlebon]

Related OSTree issue: ostreedev/ostree#1793

[cgwalters]

Also, if we shipped the .osmet file in e.g. /sysroot/fcos.osmet, then we could always support a factory reset back to the aleph image. If we wanted to support reset back to the current ostree commit we'd need a tool to fetch it out of band.

(This all said I think a lot of cases are going to be happy enough with the "ostree level" factory reset and not "wipe and replace disk image", particularly because the former can be made transactional)

[cgwalters]

As is today, it will work in many basic scenarios to just do:

$ unshare -m /bin/sh -c 'mount -o remount,rw /boot && touch /boot/ignition.firstboot'
$ reboot

A much stronger version would look like touch /run/factory-reset and we go back into the initramfs at shutdown time and do e.g. rm /var/* -rf && rsync -rlv --delete /usr/etc/ etc/.

The strongest version here of course is re-fetching and re-imaging the target disk from the initramfs, but that's even more involved.

[jlebon]

Also, if we shipped the .osmet file in e.g. /sysroot/fcos.osmet, then we could always support a factory reset back to the aleph image. If we wanted to support reset back to the current ostree commit we'd need a tool to fetch it out of band.

Truly shipping the osmet file in the image is impossible because it's a circular dependency. So it would probably instead require coreos-installer to propagate it there manually at install time. But the problem is that the osmet file is tied to the OSTree commit of the aleph version, so if the node upgraded at all, we would need to fetch the aleph OSTree commit. One thing we could easily do is have coreos-installer add a ref to it it's not GC'ed, but then every node is paying that storage cost.

[bgilbert]

$ unshare -m /bin/sh -c 'mount -o remount,rw /boot && touch /boot/ignition.firstboot'
$ reboot

This won't remove any existing customizations, so I don't think we should encourage it.

[jlebon]

A variation on #399 (comment) using kexec would be:

• download the rootfs CPIO corresponding to the version we're at from stream metadata
• concatenate base OSTree initrd with rootfs initrd and target Ignition config initrd
• kexec with base OSTree kernel + concatenated initrd and --append 'coreos.inst.install_dev=... ... etc, to reinstall on the same device

Since the rootfs CPIO includes the osmet file, no network is required during install. But air-gapped systems will need a way to obtain the rootfs initrd in the first place.

Would require some tweaks to support pointing at a local Ignition config from coreos.inst.* kargs, but if we're wrapping all this in a e.g. coreos-installer reinstall, then we could also just generate a live Ignition config and do away with coreos.inst.* kargs entirely.

[jlebon]

A variation on #399 (comment) using kexec would be:

Opened a proof of concept for this in coreos/coreos-installer#712.

[dustymabe]

We discussed this in the community meeting today.

12:04:26    dustymabe | #agreed For our factory reset capability we'll use kexec and initially
                      | require either network access or local copies of the install media to
                      | exist. In the future we may generate intall media from the existing system.
                      | We also will limit our scope to just re-installing FCOS on FCOS and
                      | disallow/discourage running it from other distros.

[lucab]

Regarding kexec usage, man 7 kernel_lockdown says:

Only validly signed binaries may be kexec'd (waived if the binary image file to be executed is vouched for by IMA appraisal).

Kernel lockdown is automatically enabled when booting with UEFI SecureBoot, so I think that in practice we'll end up coupled with the lifecycle of the Fedora kernel signing key which is embedded in the currently booted FCOS kernel.
There may be a way to load new keys for the target kernel signed by a different key, I didn't dig into that.

[cgwalters]

A new take on this in containers/bootc#137

[ericcurtin]

I notice kexec was brought up, I only started learning this recently, while kexec works great on x86 platforms, there's some embedded platforms it doesn't work on. Apple Silicon/Asahi being one for example

[ericcurtin]

As is today, it will work in many basic scenarios to just do:

$ unshare -m /bin/sh -c 'mount -o remount,rw /boot && touch /boot/ignition.firstboot'
$ reboot

A much stronger version would look like touch /run/factory-reset and we go back into the initramfs at shutdown time and do e.g. rm /var/* -rf && rsync -rlv --delete /usr/etc/ etc/.

The strongest version here of course is re-fetching and re-imaging the target disk from the initramfs, but that's even more involved.

I may look into this as it could be useful for an Automotive feature.

I think bootc will ultimately have the cleanest approach but it's hard to beat a neat tool like bootc that has an in-built installer right? 😄

How about this as an option. As hinted in to the above approach as a more involved solution...

In ostree-prepare-root check some directory in the sysroot for a file (/run/factory-reset is fine that persists reboots right?)

If that file exists, in the C code during initrd do:

rm -rf var/*
rm -rf etc
cp -r usr/etc etc

and continue boot, more complex than this of course, but the above would be the gist of things.

This wouldn't be a live factory reset of course, but live resets are different.

[jlebon]

I notice kexec was brought up, I only started learning this recently, while kexec works great on x86 platforms, there's some embedded platforms it doesn't work on. Apple Silicon/Asahi being one for example

Possibly there's kinks that need to be worked out there specific to Apple Silicon, but AFAICT it is supported on aarch64 in general. We also have a kdump test which has no arch restriction (and so runs on all the arches we currently support).

[cgwalters]

It's already possible to do a form of this via ostree admin deploy --no-merge - and the really nice thing about that is it becomes trivial to carry forward state you do want from /etc by just copying it into the new deployment's /etc.

It's also transactional.

Resetting /var...hmm; I'm not sure it needs to be part of ostree, or at least not part of ostree-prepare-root.service which already does too much. The tricky thing here is offering some mechanism to only reset certain state.

I think I'd argue that this can be done in the real root during shutdown (after we've finalized the new bootloader entry). Something we likely should do in this scenario though is rerun through the new ostree logic to restore the factory /var state. If the /var cleanup happens in the real root then arbitrary complex tooling can operate on it using the full real root userspace.

There are some other bits of state though:

• origin file
• kernel arguments (it'd be nice to retain the original install kargs for sure...)

For these...it'd definitely be nice to retain the "aleph"/originally-installed state of these.