lib/sysroot-deploy: Add experimental support for automatic early prune

[jlebon]

During the early design of FCOS and RHCOS, we chose a value of 384M
for the boot partition. This turned out to be too small: some arches
other than x86_64 have larger initrds, kernel binaries, or additional
artifacts (like device tree blobs). We'll likely bump the boot partition
size in the future, but we don't want to abandon all the nodes deployed
with the current size.[1]

Because stale entries in /boot are cleaned up after new entries are
written, there is a window in the update process during which the bootfs
temporarily must host all the (kernel, initrd) pairs for the union of
current and new deployments.

This patch determines if the bootfs is capable of holding all the
pairs. If it can't but it could hold all the pairs from just the new
deployments, the outgoing deployments (e.g. rollbacks) are deleted
before new deployments are written. This is done by updating the
bootloader in two steps to maintain atomicity.

Since this is a lot of new logic in an important section of the
code, this feature is gated for now behind an environment variable
(OSTREE_SYSROOT_OPTS=early-prune). Once we gain more experience with it,
we can consider turning it on by default.

This strategy increases the fallibility of the update system since one
would no longer be able to rollback to the previous deployment if a bug
is present in the bootloader update logic after auto-pruning. This is
however mitigated by the fact that the heuristic is opportunistic: the
rollback is pruned only if it's the only way for the system to update.

Closes: #2670

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[jlebon]

I still need to add tests for this. Prep patches split out in #2848.

[jlebon]

Now with a test!

[cgwalters]

Gave this a skim, seems sane.

[dustymabe]

During the early design of FCOS and RHCOS, we chose a value of 384M for the boot partition. This turned out to be too small: some arches other than x86_64 have larger initrds, kernel binaries, or additional artifacts (like device tree blobs). We'll likely bump the boot partition size in the future, but we don't want to abandon all the nodes deployed with the current size.[1]

Because stale entries in /boot are cleaned up after new entries are written, there is a window in the update process during which the bootfs temporarily must host all the (kernel, initrd) pairs for the union of current and new deployments.

This patch determines if the bootfs is capable of holding all the pairs. If it can't but it could hold all the pairs from just the new deployments, the outgoing deployments (e.g. rollbacks) are deleted before new deployments are written. This is done by updating the bootloader in two steps to maintain atomicity.

Since this is a lot of new logic in an important section of the code, this feature is gated for now behind an environment variable (OSTREE_EXP_AUTO_EARLY_PRUNE). Once we gain more experience with it, we can consider turning it on by default.

If we lift it out of experimental in the future, but leave it off by
default should we consider removing EXP from the name of the var
here?

Also, should we consider allowing the var to be used to turn off the
behavior (i.e. if we flip it to the the default in the future and
someone finds they don't want the behavior).

This strategy increases the fallibility of the update system since one would no longer be able to rollback to the previous deployment if a bug is present in the bootloader update logic after auto-pruning. This is however mitigated by the fact that the heuristic is opportunistic: the rollback is pruned only if it's the only way for the system to update.

So basically if you have X,Y -> Z and there is a logic problem
in Y you can't get back to X in order to update to Z? Your words
are a bit hard to decipher but this simple example makes it clear to me
(assuming the simple example is correct).

[jlebon]

If we lift it out of experimental in the future, but leave it off by default should we consider removing EXP from the name of the var here?

The idea with the EXP variable is that it's purposely temporary. If it's stabilized but left off by default, I think we'd want this controlled by e.g. a repo config knob instead. That said, I've removed the EXP wording since it could make sense even long-term to have a variable override for it to get out of a pickle (given that e.g. repo configs are unmanaged, but systemd dropins aren't).

Also, should we consider allowing the var to be used to turn off the behavior (i.e. if we flip it to the the default in the future and someone finds they don't want the behavior).

Since it's currently not the default, this would be decided at the time we do decide to flip it to be the default. We could e.g. keep the variable around and slightly tweak the existing checks. Related to this, I've updated the check so that we not only skip it if the variable is defined but empty, but also if it's set to 0.

So basically if you have X,Y -> Z and there is a logic problem in Y you can't get back to X in order to update to Z? Your words are a bit hard to decipher but this simple example makes it clear to me (assuming the simple example is correct).

I didn't want to dig too much into this in the commit message because indeed it's a bit subtle and would make it even longer. I linked to #2670 (comment) instead which with the following comments should help I think.

[cgwalters]

A few minor changes

[jlebon]

Updated for comments!

[jlebon]

I forgot to update the commit message on this to reference the new environment variable knob (OSTREE_SYSROOT_OPTS=early-prune) rather than the initial OSTREE_ENABLE_AUTO_EARLY_PRUNE one. I've updated the description, but it'll forever be in git history, lying to readers... 😢