Initial support for live PXE and ISO images #155

bgilbert · 2019-09-05T06:32:48Z

Deferred to follow-up PRs:

Functioning with SELinux enabled
Allowing users to provide an Ignition config via a custom initrd image
Automatic login in the ISO image

Other known issues:

dracut passes the ostree path to ostree-prepare-root by bind-mounting over /proc/cmdline
rpm-ostreed refuses to start
ISO kargs are not automatically synced from the BLS configs

Fixes #115.

- Change labels - Use short bootloader timeout - Disable installer - Enable Ignition, defaulting to metal platform - Drop nomodeset karg

cgwalters

Overall looks good! I'd like to think a bit harder about the "API" around which live mounts are detected. I feel like e.g. /run/ostree-booted has worked well; why not have /run/ostree-live or so and use ConditionPathExists=/run/ostree-live rather than ConditionPathIsReadWrite=/?

It's funny because just last night I was working on ostreedev/ostree#1265
and while that happens to leave / as technically rw...I am not sure I really want to encode that in many systemd units permanently.

BTW...there is also https://www.freedesktop.org/software/systemd/man/systemd-volatile-root.service.html which was clearly intended for something like this, but it doesn't deal with /etc for example.

cgwalters · 2019-09-05T13:21:01Z

overlay.d/05core/usr/lib/dracut/modules.d/20live/module-setup.sh

@@ -0,0 +1,4 @@
+install() {
+    inst_script "$moddir/is-live-image.sh" \
+        "/usr/bin/is-live-image"


I know this is just the initramfs but I am not a fan of "/usr/bin pollution". /usr/libexec/coreos-is-live-image ?

This is an API with ignition-dracut, which is distro-independent, so it shouldn't be coreos-. We could use /usr/libexec but nothing else in the initrd does (except coreos-installer), and /usr/bin pollution is rampant in the initrd, so I'm inclined to leave it.

cgwalters · 2019-09-05T13:24:52Z

overlay.d/05core/usr/lib/dracut/modules.d/20live/coreos-populate-writable.service

@@ -0,0 +1,9 @@
+[Unit]
+DefaultDependencies=false
+RequiresMountsFor=/writable


Don't we preserve /run from the initramfs? If so, let's put stuff in /run/coreos-live-writable or so?

We do. I thought it'd be cleaner to have a dedicated tmpfs for user-created state. Also, this avoids exposing the overlayfs upperdir and workdir within the directory structure.

overlay.d/05core/usr/lib/dracut/modules.d/20live/coreos-populate-writable.service

cgwalters · 2019-09-05T13:27:15Z

overlay.d/05core/usr/lib/dracut/modules.d/20live/live-generator

+    add_requires sysroot-var.mount
+
+    mkdir -p "${UNIT_DIR}/ostree-prepare-root.service.d"
+    cat > "${UNIT_DIR}/ostree-prepare-root.service.d/10-live.conf" <<EOF


I'm OK with this, though I'd also like to consider standardizing support for this in ostree itself at some point.

Yeah, that'd be good. This is meant as a starting point.

It's definitely hacky to bind mount over /proc/cmdline but is a creative way to get this working now. Can you open an RFE against ostree to support this natively and copy the URL to the issue in this dropin unit as well as a comment in the ostree-cmdline.sh script?

overlay.d/05core/usr/lib/systemd/system-generators/coreos-boot-mount-generator

cgwalters · 2019-09-05T13:32:01Z

live/EFI/fedora/grub.cfg

@@ -28,6 +28,6 @@ set timeout=1

 ### BEGIN /etc/grub.d/10_linux ###
 menuentry 'Fedora CoreOS (Live)' --class fedora --class gnu-linux --class gnu --class os {
-	linux /images/vmlinuz rd.neednet=1 ip=dhcp ignition.firstboot ignition.platform.id=metal
+	linux /images/vmlinuz mitigations=auto,nosmt rd.neednet=1 ip=dhcp ignition.firstboot ignition.platform.id=metal


Hmm. It feels cleaner if we pulled this stuff out of an existing disk image. But OK for now.

bgilbert · 2019-09-05T17:27:54Z

I feel like e.g. /run/ostree-booted has worked well; why not have /run/ostree-live or so and use ConditionPathExists=/run/ostree-live rather than ConditionPathIsReadWrite=/?

I'm okay with that. ConditionPathIsReadWrite doesn't feel 💯 to me.

ignition-dracut will invoke this to determine whether to enable units that require a boot disk.

bgilbert · 2019-09-07T00:14:21Z

Updated.

lucab · 2019-09-09T08:11:44Z

Just for clarity, which component is going to create /run/ostree-live and at what point of the build (or boot) flow?

bgilbert · 2019-09-09T09:40:31Z

@lucab It's currently being done at boot time, in live-generator in the initramfs.

cgwalters · 2019-09-09T13:16:34Z

This all LGTM!

bgilbert · 2019-09-09T18:42:41Z

To test:

Pull in this PR and Build live PXE and ISO images coreos-assembler#738.
Copy the contents of Conditionalize units which require a boot disk ignition-dracut#102 into overrides/rootfs.
coreos-assembler build && coreos-assembler buildextend-live
Test:

# PXE
qemu-system-x86_64 -machine accel=kvm -m 2048 \
    -netdev user,id=eth0,hostfwd=tcp::2222-:22,hostname="live" -device virtio-net-pci,netdev=eth0 \
    -fw_cfg name=opt/com.coreos/config,file=$HOME/ssh-key.ign \
    -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0 \
    -kernel fedora-coreos-30.*-live-kernel \
    -initrd fedora-coreos-30.*-live-initramfs.img \
    -append 'rd.neednet=1 ip=dhcp ignition.firstboot ignition.platform.id=qemu enforcing=0'`
# ISO
qemu-system-x86_64 -machine accel=kvm -m 2048 -cdrom fedora-coreos-30.*.iso \
    -netdev user,id=eth0,hostfwd=tcp::2222-:22,hostname="live" -device virtio-net-pci,netdev=eth0 \
    -fw_cfg name=opt/com.coreos/config,file=$HOME/ssh-key.ign \
    -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0

For ISO boot, manually edit the kargs to specify ignition.platform.id=qemu enforcing=0.

cgwalters · 2019-09-11T12:27:54Z

I'm OK merging this as is; do you think someone else needs to test before merging to master?

SerialVelocity · 2019-09-11T12:47:20Z

overlay.d/05core/usr/lib/dracut/modules.d/20live/writable.mount

+Where=/writable
+Type=tmpfs
+Options=mode=0700
+EOF


Probably want to remove this line before merging. Pretty sure it is leftover from when you extracted it out from a cat <<EOF

Great catch, thanks! Updated.

dustymabe · 2019-09-11T15:17:47Z

I'm OK merging this as is; do you think someone else needs to test before merging to master?

i'm running through reviews/tests today

dustymabe

some initial comments. I'll probably have some more tomorrow when I finally test the artifacts. Good news is they did build locally for me!

dustymabe · 2019-09-13T01:31:17Z

overlay.d/05core/usr/lib/dracut/modules.d/20live/live-generator

+Type=squashfs
+EOF
+
+cat >"${UNIT_DIR}/sysroot-etc.mount" <<EOF


dustymabe · 2019-09-13T01:31:25Z

overlay.d/05core/usr/lib/dracut/modules.d/20live/live-generator

+Options=lowerdir=/sysroot/etc,upperdir=/writable/etc/upper,workdir=/writable/etc/work,redirect_dir=on,index=on,xino=on
+EOF
+
+cat >"${UNIT_DIR}/sysroot-var.mount" <<EOF


dustymabe · 2019-09-13T01:46:56Z

overlay.d/05core/usr/lib/dracut/modules.d/20live/ostree-cmdline.sh

+set -euo pipefail
+
+case "${1:-unset}" in
+start)


indent these cases?

dustymabe · 2019-09-13T01:51:52Z

overlay.d/05core/usr/lib/dracut/modules.d/20live/live-generator

+    add_requires sysroot-var.mount
+
+    mkdir -p "${UNIT_DIR}/ostree-prepare-root.service.d"
+    cat > "${UNIT_DIR}/ostree-prepare-root.service.d/10-live.conf" <<EOF


It's definitely hacky to bind mount over /proc/cmdline but is a creative way to get this working now. Can you open an RFE against ostree to support this natively and copy the URL to the issue in this dropin unit as well as a comment in the ostree-cmdline.sh script?

dustymabe · 2019-09-13T02:12:54Z

overlay.d/05core/usr/lib/systemd/system-generators/coreos-boot-mount-generator

+Where=/boot
+EOF
+
+    # Mount /boot/efi only on UEFI platforms, as well as x86_64 BIOS


can we add a comment here on why we want this behavior on these specific cases

overlay.d/05core/usr/lib/systemd/system-generators/coreos-boot-mount-generator

dustymabe · 2019-09-13T02:24:25Z

overlay.d/05core/usr/lib/systemd/system-generators/coreos-boot-mount-generator

+    ln -sf "../${name}" "${wants_dir}/${name}"
+}
+
+# Don't create mount units for /boot or /boot/efi on live systems


it would be nice if we add a comment here about not being able to use ConditionPathExists /run/ostree-live. You have this comment in your commit message, but would be nice to have it here.

dustymabe · 2019-09-13T02:29:29Z

live/zipl.prm

@@ -0,0 +1 @@
+mitigations=auto,nosmt rd.neednet=1 ip=dhcp ignition.firstboot ignition.platform.id=metal


regarding the mitigations=auto,nosmt (the last commit in this series) it seems like this is going to be a bit of a maintenance burden because we store that information currently in image.yaml can we pull those args from there?

You mean just the extra-kargs? We can, but there are also args coming from other places (ignition.platform.id, console, the temporary networking args, rootflags, etc). We could read them out of the BLS config and filter out the undesired ones, but I thought that might be more trouble than it's worth. Thoughts?

We could read them out of the BLS config and filter out the undesired ones, but I thought that might be more trouble than it's worth. Thoughts?

I like that, depending on how much trouble it really is. Basically we'd maintain a blacklist in a single place vs duplicated kernel args in various places.

OK. Can I queue that for a followup PR?

dustymabe · 2019-09-14T19:33:27Z

it looks like the selinux issues are related to files getting labeled with the tmpfs_t type probably because they are getting fiddled with under the /writable tmpfs at some point?

If /root.squashfs exists in the initramfs, do the following: 1. Mount /root.squashfs on /sysroot 2. Locate the ostree deployment, add a corresponding ostree= argument to /proc/cmdline by bind-mounting over it, run ostree-prepare-root.service to set up the deployment, and unmount /proc/cmdline 3. Mount a tmpfs on /writable 4. Create needed directories in /writable 5. Create a writable overlay on /sysroot/etc, with /sysroot/etc as lowerdir and upper/work dirs in /writable 6. Bind-mount /writable/var to /sysroot/var 7. Run coreos-populate-var.service to populate /sysroot/var Disable mounting of stateroot /var on live systems, since that's handled by step 6.

There's no on-disk filesystem to resize.

Conditionalizing the units won't work here because conditions don't affect the dependency on the underlying device unit. Enable boot.mount on non-PXE boots. Enable boot-efi.mount on non-PXE boots on platforms (x86_64 and aarch64) where we have an EFI partition, whether the /boot/efi mountpoint exists or not. Also make the mount units WantedBy local-fs.target, rather than RequiredBy, to prevent boot failures if the mount fails.

We enable it in non-live builds.

bgilbert · 2019-09-15T04:13:42Z

Updated.

dustymabe · 2019-09-15T16:52:21Z

OK. I ran through and tested the live PXE, live ISO, and installer ISO with these changes. We have some outstanding comments but they are mostly small fixup items and some of them will lead to followup work (rather than work to do in this PR). We should be able to address the remaining small comments on Monday and open a few issues for followup work (maybe others can pick up some of the tasks) and then get this in!

Nice work @bgilbert!

bgilbert · 2019-09-15T18:32:19Z

@dustymabe Thanks for the reviews and testing!

You're right about SELinux. I have a followup commit that fixes that via relabel-extra.d, but it depends on both selinux-policy-3.14.3-46.fc30 and systemd/systemd#13524.

As far as I know, all of the feedback on this PR set has now been addressed, other than missing functionality that can be handled in a followup. @dustymabe, is there something I've missed? I'd like to get this set of PRs merged to clear the way for the next set.

dustymabe

LGTM - I've enumerated some followup items below. Should we open up a few issues to track anything that doesn't already have one?

Initial support for live PXE and ISO images #155 (comment)
- pull kernel args for live images from common location as other disk images, possibly blacklist ones we know we don't want for lives
Initial support for live PXE and ISO images #155 (comment)
- enable selinux for live images
rpm-ostreed refuses to start
- Make ostree aware of live environments ostreedev/ostree#1921
Initial support for live PXE and ISO images #155 (comment)
- Provide a way to override the ostree= kernel argument ostreedev/ostree#1920
- dracut passes the ostree path to ostree-prepare-root by bind-mounting over /proc/cmdline rpm-ostreed refuses to start

dustymabe · 2019-09-16T17:36:43Z

One other thing that I thought of this weekend: It would probably be really nice to have a high level design doc for how this all works since it can be hard to stitch all of the pieces together from the various repos. It's optional of course, but a design doc in the fcos tracker repo (or somewhere) might be helpful to us all in the future.

bgilbert · 2019-09-16T19:56:42Z

pull kernel args for live images from common location as other disk images, possibly blacklist ones we know we don't want for lives

Should be done this week.

enable selinux for live images

Coming in followup PR.

bgilbert added 2 commits August 29, 2019 05:02

live: copy boot files from installer

bf49343

live: update for live images

7c866d2

- Change labels - Use short bootloader timeout - Disable installer - Enable Ignition, defaulting to metal platform - Drop nomodeset karg

cgwalters reviewed Sep 5, 2019

View reviewed changes

overlay/05core: add hook script to detect live system

88aca0a

ignition-dracut will invoke this to determine whether to enable units that require a boot disk.

lucab mentioned this pull request Sep 9, 2019

dist/systemd: don't start Zincati if booted as a live system coreos/zincati#126

Merged

dustymabe self-requested a review September 9, 2019 15:18

cgwalters mentioned this pull request Sep 9, 2019

DNM: overlay/systemd/boot-efi.mount: use efivars #106

Closed

SerialVelocity reviewed Sep 11, 2019

View reviewed changes

dustymabe reviewed Sep 13, 2019

View reviewed changes

bgilbert added 4 commits September 15, 2019 00:04

overlay/05core: skip coreos-growpart.service on live systems

4a8a2e2

There's no on-disk filesystem to resize.

live: add mitigations=auto,smt karg

1dfa80f

We enable it in non-live builds.

dustymabe approved these changes Sep 16, 2019

View reviewed changes

bgilbert merged commit d1b0415 into coreos:testing-devel Sep 16, 2019

bgilbert deleted the live branch September 16, 2019 19:57

SerialVelocity mentioned this pull request Sep 19, 2019

Don't relabel across mounts coreos/fedora-coreos-tracker#223

Closed

jlebon mentioned this pull request Dec 5, 2019

testing: failed to start rpm-ostree daemon after auto-update to 31.20191127.1 coreos/fedora-coreos-tracker#322

Closed

dustymabe pushed a commit to jbtrystram/fedora-coreos-config that referenced this pull request Apr 19, 2024

note use of FCOS by quay.io (coreos#155)

f537eb6

		@@ -0,0 +1 @@
		mitigations=auto,nosmt rd.neednet=1 ip=dhcp ignition.firstboot ignition.platform.id=metal

Initial support for live PXE and ISO images #155

Initial support for live PXE and ISO images #155

Conversation

bgilbert commented Sep 5, 2019 • edited Loading

cgwalters left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgilbert commented Sep 5, 2019

bgilbert commented Sep 7, 2019

lucab commented Sep 9, 2019

bgilbert commented Sep 9, 2019

cgwalters commented Sep 9, 2019

bgilbert commented Sep 9, 2019

cgwalters commented Sep 11, 2019

SerialVelocity Sep 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dustymabe commented Sep 11, 2019

dustymabe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dustymabe commented Sep 14, 2019

bgilbert commented Sep 15, 2019

dustymabe commented Sep 15, 2019 • edited Loading

bgilbert commented Sep 15, 2019

dustymabe left a comment

Choose a reason for hiding this comment

dustymabe commented Sep 16, 2019

bgilbert commented Sep 16, 2019

bgilbert commented Sep 5, 2019 •

edited

Loading

SerialVelocity Sep 11, 2019 •

edited

Loading

dustymabe commented Sep 15, 2019 •

edited

Loading