-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add factory reset capability #399
Comments
Discussed in the community meeting today.
There was a general sense that the feature would be nice, but there was also concern about implementation complexity depending on the approach taken. |
Note with the stage 2 approach, as long as the stage 2 binary is the initrd we append to the base one, it should contain both the osmet file and the root squashfs, so it should be possible to do the re-install completely offline. |
Related OSTree issue: ostreedev/ostree#1793 |
Also, if we shipped the (This all said I think a lot of cases are going to be happy enough with the "ostree level" factory reset and not "wipe and replace disk image", particularly because the former can be made transactional) |
As is today, it will work in many basic scenarios to just do:
A much stronger version would look like The strongest version here of course is re-fetching and re-imaging the target disk from the initramfs, but that's even more involved. |
Truly shipping the osmet file in the image is impossible because it's a circular dependency. So it would probably instead require coreos-installer to propagate it there manually at install time. But the problem is that the osmet file is tied to the OSTree commit of the aleph version, so if the node upgraded at all, we would need to fetch the aleph OSTree commit. One thing we could easily do is have coreos-installer add a ref to it it's not GC'ed, but then every node is paying that storage cost. |
This won't remove any existing customizations, so I don't think we should encourage it. |
A variation on #399 (comment) using
Since the rootfs CPIO includes the osmet file, no network is required during install. But air-gapped systems will need a way to obtain the rootfs initrd in the first place. Would require some tweaks to support pointing at a local Ignition config from |
Opened a proof of concept for this in coreos/coreos-installer#712. |
We discussed this in the community meeting today.
|
Regarding
Kernel lockdown is automatically enabled when booting with UEFI SecureBoot, so I think that in practice we'll end up coupled with the lifecycle of the |
Ironically our support for `--replace-mode=alongside` breaks when we're targeting an already extant ostree host, because when we first blow away the `/boot` directory, this means the ostree stack loses its knowledge that we're in a booted deployment, and will attempt to GC it... ostreedev/ostree-rs-ext@8fa019b is a key part of the fix for that. However, a notable improvement we can do here is to grow this whole thing into a real "factory reset" mode, and this will be a compelling answer to coreos/fedora-coreos-tracker#399 To implement this though we need to support configuring the stateroot and not just hardcode `default`.
Ironically our support for `--replace-mode=alongside` breaks when we're targeting an already extant ostree host, because when we first blow away the `/boot` directory, this means the ostree stack loses its knowledge that we're in a booted deployment, and will attempt to GC it... ostreedev/ostree-rs-ext@8fa019b is a key part of the fix for that. However, a notable improvement we can do here is to grow this whole thing into a real "factory reset" mode, and this will be a compelling answer to coreos/fedora-coreos-tracker#399 To implement this though we need to support configuring the stateroot and not just hardcode `default`. Signed-off-by: Colin Walters <walters@verbum.org>
Ironically our support for `--replace-mode=alongside` breaks when we're targeting an already extant ostree host, because when we first blow away the `/boot` directory, this means the ostree stack loses its knowledge that we're in a booted deployment, and will attempt to GC it... ostreedev/ostree-rs-ext@8fa019b is a key part of the fix for that. However, a notable improvement we can do here is to grow this whole thing into a real "factory reset" mode, and this will be a compelling answer to coreos/fedora-coreos-tracker#399 To implement this though we need to support configuring the stateroot and not just hardcode `default`. Signed-off-by: Colin Walters <walters@verbum.org>
A new take on this in containers/bootc#137 |
Ironically our support for `--replace-mode=alongside` breaks when we're targeting an already extant ostree host, because when we first blow away the `/boot` directory, this means the ostree stack loses its knowledge that we're in a booted deployment, and will attempt to GC it... ostreedev/ostree-rs-ext@8fa019b is a key part of the fix for that. However, a notable improvement we can do here is to grow this whole thing into a real "factory reset" mode, and this will be a compelling answer to coreos/fedora-coreos-tracker#399 To implement this though we need to support configuring the stateroot and not just hardcode `default`. Signed-off-by: Colin Walters <walters@verbum.org>
I notice kexec was brought up, I only started learning this recently, while kexec works great on x86 platforms, there's some embedded platforms it doesn't work on. Apple Silicon/Asahi being one for example |
I may look into this as it could be useful for an Automotive feature. I think bootc will ultimately have the cleanest approach but it's hard to beat a neat tool like bootc that has an in-built installer right? 😄 How about this as an option. As hinted in to the above approach as a more involved solution... In ostree-prepare-root check some directory in the sysroot for a file (/run/factory-reset is fine that persists reboots right?) If that file exists, in the C code during initrd do:
and continue boot, more complex than this of course, but the above would be the gist of things. This wouldn't be a live factory reset of course, but live resets are different. |
Possibly there's kinks that need to be worked out there specific to Apple Silicon, but AFAICT it is supported on aarch64 in general. We also have a kdump test which has no arch restriction (and so runs on all the arches we currently support). |
It's already possible to do a form of this via It's also transactional. Resetting I think I'd argue that this can be done in the real root during shutdown (after we've finalized the new bootloader entry). Something we likely should do in this scenario though is rerun through the new ostree logic to restore the factory There are some other bits of state though:
For these...it'd definitely be nice to retain the "aleph"/originally-installed state of these. |
Under the immutable infrastructure model, we encourage users to handle config changes by reprovisioning their systems, rather than using configuration management or manually modifying the running system. This is okay in virtualized setups or in the cloud, and is also okay on bare metal where PXE infrastructure or IPMI-assisted ISO boot is available. But on single-node bare metal setups without external infrastructure, manual reinstall is impractical. (See e.g. this discussion.)
For this use case, it might help to have a command which receives an Ignition config to use for reprovisioning, copies it into
/boot
, sets first-boot kargs, and reboots. Then the initramfs, before running the usual first-boot steps, would need to restore the system to a pristine state. That process might look a lot like #94, or might look more like clearing/var
and resetting/etc
from/usr/etc
. If those are in non-standard locations, perhaps the user-mode command could add kargs which tell the initramfs where to find them.Because we'd be deleting user customizations before rerunning Ignition, this approach wouldn't[*] get into unsupported territory of using Ignition for configuration management. It would slightly complicate OS maintenance, though. At present we can freely change the Ignition initrd glue in ways that are incompatible with old installs (e.g. by changing first-boot kargs) because old installs will never run Ignition again. Supporting factory reset of old nodes would presumably add some additional constraints.
[*] Presumably it'd be infeasible to delete every possible user customization, so that statement might be optimistic.
The text was updated successfully, but these errors were encountered: