WIP: Redeploy rootfs

[cgwalters]

This is a quick hack to add a 31ignition-ostree module that can take care of redeploying the rootfs between ignition-disks.service and ignition-mount.service.

We obviously only want to run this if the user has provided an Ignition config that replaces the root labeled filesystem. Doing so I think would require adding something like a separate ignition-parse.service stage that touches a stamp file /run/ignition-rootfs-replaced or so, and these services key off that?

I'm working on this because openshift/enhancements#15 needs it.

[cgwalters]

OK this turns out to block on the SELinux/initramfs issue: #79 (comment)

Everything ends up unlabeled_t because mv/cp see policy isn't loaded, but we need to force them to preserve contexts.

Also annoying is that cp/mv don't exit on the first error. It's tempting to implement this in rpm-ostree (in Rust and not shell script) as something like rpm-ostree coreos-internals redeploy-rootfs.

[jlebon]

More specifically, it blocks on https://lore.kernel.org/selinux/20190912133007.27545-1-jlebon@redhat.com/T/#u. The third paragraph in the commit message covers this exact use case. :)

Without that, we're basically stuck between two bad options (loading the policy in the initrd or doing some kind of clever hack so we can redeploy post-pivot but then we'd still have to reboot).

[darkmuggle]

I'm working on this because openshift/enhancements#15 needs it.

To clarify: this will be useful, and I think we should include it, but the enhancement request does not need it. The enhancement request specifically states the root filesystem will be written in a null-cipher encrypted root filesystem, for the simple reason that Ignition is lacking support for setting up LUKS volumes.

Where I see this work being immediately relevant is that if a user does not want encryption, then the LUKS container can be replaced. Longer-term, once Ignition supports setting up LUKS, then we can have a default Ignition configuration for setting up LUKS and use this Dracut module to rewrite the filesystem.

[darkmuggle]

Sorry.... @cgwalters that was entirely an accidental closure.

[cgwalters]

The enhancement request specifically states the root filesystem will be written in a null-cipher encrypted root filesystem, for the simple reason that Ignition is lacking support for setting up LUKS volumes.

Sure, but I don't like the idea of shipping that versus solving the full general problem. We already had to close PRs like coreos/coreos-installer#48 because we don't have this.

[cgwalters]

@ajeddeloh When you get a few can you look over this from an architecture perspective? Particularly bits like:

How do we hook in a "post-parse pre-disks" Ignition hook for this to run?

Relatedly, do you think it's better if Ignition gained an optional ostree integration (and had this code, hence no direct hook necessary) or if (rpm-)ostree gained this code and we had the hook?

[ajeddeloh]

In general I'm in favor of this and the post-parse-pre-disks hook. This is dependent on splitting parsing into it's own stage, but I've been wanting to do that anyway (this has also been called "stage 0"). I'm not sure how coupled to Ignition it ought to be. The simplest way would be to say "is there a filesystem in the config with "label": "root". It could give false positives in some cases but shouldn't give false negatives. That is simple enough we could do it with just jq, I think an Ignition stage would be excessive for that. If we end up needing something more complex we can also switch to an Ignition stage.

The only concern with false positives I have is it's a performance penalty. We could reduce it a fair bit by not blowing away the rootfs prematurely (looks like that might just be for testing in this pr?) and checking if there are ostree files already before redeploying. This still has the save penalty, but not the restore.

[cgwalters]

I don't quite understand your suggestion; I was thinking of a ignition-fetch.service which ran a fetch stage which just wrote /run/ignition.json - from there we could parse it with jq as you say.

In general I'm in favor of this and the post-parse-pre-disks hook

When I was saying "hook" I was thinking of how we use systemd units today to separate things out. Were you thinking something more like a build-time option for Ignition that ran a specified binary? Or an /etc/ignition/post-fetch-hooks.d?

[ajeddeloh]

I was thinking of a ignition-fetch.service which ran a fetch stage which just wrote /run/ignition.json - from there we could parse it with jq as you say.

When I was saying "hook" I was thinking of how we use systemd units today to separate things out.

That's exactly what I was thinking of, we're on the same page.

Were you thinking something more like a build-time option for Ignition that ran a specified binary?

Only if jq proves insufficient

Or an /etc/ignition/post-fetch-hooks.d?

Wouldn't be needed with a fetch stage (can use systemd units)

[cgwalters]

OK cool split that out into #110 and wrote some jq and code to try to conditionalize this.

Also here's the rootfs.fcct I've been testing out:

# Example of redeploying the rootfs as ext4
variant: fcos
version: 1.0.0
storage:
  filesystems:
    - device: /dev/disk/by-partlabel/root
      format: ext4
      wipe_filesystem: true
      label: root

[cgwalters]

Moved to coreos/fedora-coreos-config#184