Build ZFS sysext with each release #1742

jepio · 2024-03-11T10:32:15Z

Build ZFS sysext with each release

Add a ZFS sysext that is built with every Flatcar release and can be enabled through the /etc/flatcar/enabled-sysext.conf mechanism that @pothos added. Since it provides a kernel module, it is tightly coupled to the Flatcar release. To solve issues with etc configs and udev rules needing to be run after the sysext is loaded, I created a tmpfiles.d that sets up the required symlinks and ordering between sysext.service and udev.service.

Following @pothos advice, i've added a step for building the sysexts after the image build. This DEFAULT_SYSEXTS can be extended to add more sysexts.

There are also some other improvements that I've added:

sysext building can happen in parallel
SOURCE_DATE_EPOCH is set for sysext build. I've taken the timestamp of the /usr/lib/os-release file as the source of the timestamp.
coreos-kernel now provides a better module build environment in /lib/modules/.../build so that the ZFS mod can be built and which should simplify building nvidia-drivers.
app-alternatives/awk is now pulled into the image instead of gawk.
update_ebuild --portage=rsync now works for eclasses

Pulls in this:

initrd-setup-root-after-ignition: Ensure /etc/flatcar/sysext exists bootengine#89

How to use

(still need to try the end to end flow once this is on bincache)

./build_image
./build_image sysext
scp flatcar-zfs.raw flatcar:/etc/extensions/

[ describe what reviewers need to do in order to validate this PR ]

Testing done

[Describe the testing you have done before submitting this PR. Please include both the commands you issued as well as the output you got.]

Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

build_library/prod_image_util.sh

build_library/sysext_mangle_flatcar-zfs

pothos · 2024-03-11T11:29:14Z

Overall this looks very nice :)
For release publishing we already have the required logic in download_payloads and generate_payload so I hope that it works smooth and we don't run into bugs while we exercise this path for the first time.

github-actions · 2024-03-11T11:29:25Z

Build action triggered: https://github.com/flatcar/scripts/actions/runs/8389540829

This function is meant to prebuild certain sysexts to be released along with each release. These will not be built into the image, but instead can be fetched by the user on demand. The command to build sysexts would be: ./build_image prod sysext Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

by skipping the check for an existing image directory if we're not building an image. This makes './build_image sysext' work. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

For eclasses we need to skip the mkdir and trailing slash. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

build/source, which is accidentally an empty directory, needs to be a symlink so that Gentoo kmod ebuilds can build when setting KERNEL_DIR=/lib/modules/.../build. They detect the proper layout with the symlink. The other issue is building the Nvidia driver with SYSSRC=/lib/modules/.../build. This works on Ubuntu but fails on Flatcar. Ubuntus build directory contains symlinks to includes from the source tree so recreate the same layout. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

We explicitly install gawk but our profile explicitly forces the gawk useflag for app-alternatives/awk. Some packages, like zfs, depend on app-alternatives/awk and it also installs the awk -> gawk that the gawk ebuild creates through pkg_postinst. So switch to app-alternatives/awk to make the implementation cleaner. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

and a package.provided entry for a dependency that we don't need. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

From Gentoo commit df182f2891606e757be3e8406a69f4a0e53ee324. Also import dist-kernel-utils.eclass. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

The zfs-kmod ebuild needs KERNEL_DIR to point to the correct /lib/modules directory. The zfs ebuild installs two systemd unit masks to /usr/lib/systemd which result in "dangling symlink" errors during the image build. These systemd unit masks are only necessary for old Ubuntu systems that have sysv-init-systemd compat wrappers. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

We need zfs build as a board package so that we can provide a zfs sysext. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

There are two challenges with the sysext: it needs config files in /etc and it needs udev rules for mounting during boot to work. The etc files are placed in the standard flatcar etc overlay path but the overlay is mounted from the initrd. So instead, we create a tmpfiles.d rule that symlinks the best important files over. For the udev issue, we create a drop-in in /etc that ensures udev runs after systemd-sysext. We also can't rely on systemd presets to work, so instead parse the preset file and statically create the service dependencies. For the primary zfs.target we rely on an Upholds entry. Users can still disabled unwanted services if they want. We also removed unnecessary files: - development files - initramfs related scripts Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

This allows it to be sourced from other scripts and used e.g. in generating an image changes summary. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

jepio · 2024-03-11T12:54:40Z

Had to force push because i forgot to add zfs to board-packages.

Gentoo's linux-info.eclass tries to check the kernel version by including /lib/modules/.../source/Makefile and printing $(VERSION). It unsets ARCH= before doing this (no clue why) which works with a full source tree but not with our minimized one. To fix this we need to archive arch/x86/Makefile also for arm64. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

jepio · 2024-03-12T08:33:00Z

@pothos: still need to fix something in bootengine and this looks slightly weird:

core@localhost ~ $ systemd-sysext status
HIERARCHY EXTENSIONS         SINCE
/opt      none               -
/usr      containerd-flatcar Tue 2024-03-12 08:30:52 UTC
          docker-flatcar
          flatcar-zfs
          oem-qemu

jepio · 2024-03-12T08:45:13Z

This is my test butane json:

variant: flatcar
version: 1.0.0
storage:
  files:
  - path: /etc/flatcar/enabled-sysext.conf
    contents:
      inline: |
        zfs
  directories:
  - path: /etc/flatcar/sysext
systemd:
  units:
  - name: zpool-create.service
    enabled: true
    contents: |
      [Unit]
      ConditionFirstBoot=1
      Before=first-boot-complete.target
      Wants=first-boot-complete.target

      [Service]
      ExecStart=zpool create tank /dev/vdb

      [Install]
      WantedBy=multi-user.target

Fix a missing mkdir of /sysroot/etc/flatcar/sysext from the initrd before attempting to move sysext files there. Also explicitly omit dracut zfs module from initrd generation. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

jepio · 2024-03-13T17:15:08Z

@pothos want to have this in the nearest alpha?

pothos · 2024-03-13T17:28:42Z

Would be good because the bootengine PR is already merged

jepio · 2024-03-13T18:34:13Z

@pothos pushed a changelog, if you have a suggestion you can commit it directly.

@sayanchowdhury lets merge this before building alpha.

changelog/changes/2024-03-13-zfs.md

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

pothos · 2024-03-13T22:24:48Z

Thanks a lot 🚀

abuisine · 2024-03-22T09:46:54Z

Cool to see this upstreamed !

jepio · 2024-03-22T11:23:07Z

Cool to see this upstreamed !

hey @abuisine, I was planning to write you to see if it works for your usecase. Still need to add docs, but mantle contains the test case we now have in our CI infra with butane examples, see here:
https://github.com/flatcar/mantle/blob/flatcar-master/kola/tests/sysext/zfs.go#L19-L116

abuisine · 2024-03-22T12:11:27Z

We'll make a bunch of tests next week for sure, will keep you posted

coreyjewett · 2024-03-29T09:24:20Z

Thanks @jepio! I was previously using the enix modules, but never got them to work right at boot. I switched to the alpha channel and 3913.0.0 loaded my pool as expected right after reboot. 🎉

jepio added 2 commits March 11, 2024 10:18

build_sysext: Use parallel jobs

fa41889

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

build_sysext: Set SOURCE_DATE_EPOCH for reproducibility

54c8545

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

jepio requested a review from a team March 11, 2024 10:32

jepio temporarily deployed to development March 11, 2024 10:32 — with GitHub Actions Inactive

pothos reviewed Mar 11, 2024

View reviewed changes

build_library/prod_image_util.sh Outdated Show resolved Hide resolved

pothos reviewed Mar 11, 2024

View reviewed changes

build_library/prod_image_util.sh Outdated Show resolved Hide resolved

pothos reviewed Mar 11, 2024

View reviewed changes

build_library/sysext_mangle_flatcar-zfs Outdated Show resolved Hide resolved

tormath1 mentioned this pull request Mar 11, 2024

experimental: add built-in Incus sysext image #1655

Draft

7 tasks

jepio added 13 commits March 11, 2024 11:57

build_image: Allow building sysexts for an existing image

1bb108e

by skipping the check for an existing image directory if we're not building an image. This makes './build_image sysext' work. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

update_ebuilds: Fix support for rsync of eclass

601adeb

For eclasses we need to skip the mkdir and trailing slash. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

profiles/coreos: Add zfs useflags

a4da571

and a package.provided entry for a dependency that we don't need. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

portage-stable: Add zfs & zfs-kmod ebuilds

8e1b151

From Gentoo commit df182f2891606e757be3e8406a69f4a0e53ee324. Also import dist-kernel-utils.eclass. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

.github: Add zfs ebuilds and related eclasses to sync list

bda8148

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

board-packages: Add zfs dep

6ccd5b6

We need zfs build as a board package so that we can provide a zfs sysext. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

ci: Build sysexts together with image

53e1649

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

build_library: Place EXTRA_SYSEXTS in a separate file

7929434

This allows it to be sourced from other scripts and used e.g. in generating an image changes summary. Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

jepio force-pushed the jepio/zfs-sysext branch from 0f1d69d to 7929434 Compare March 11, 2024 12:51

jepio temporarily deployed to development March 11, 2024 12:52 — with GitHub Actions Inactive

jepio had a problem deploying to development March 11, 2024 17:36 — with GitHub Actions Error

jepio force-pushed the jepio/zfs-sysext branch from 7dee622 to 420de9d Compare March 11, 2024 17:40

jepio temporarily deployed to development March 11, 2024 17:40 — with GitHub Actions Inactive

jepio force-pushed the jepio/zfs-sysext branch from 420de9d to 1ec40b2 Compare March 11, 2024 20:03

jepio temporarily deployed to development March 11, 2024 20:03 — with GitHub Actions Inactive

jepio temporarily deployed to development March 12, 2024 14:08 — with GitHub Actions Inactive

pothos approved these changes Mar 13, 2024

View reviewed changes

jepio force-pushed the jepio/zfs-sysext branch from 1916d79 to 333e722 Compare March 13, 2024 18:28

jepio had a problem deploying to development March 13, 2024 18:28 — with GitHub Actions Error

jepio had a problem deploying to development March 13, 2024 18:33 — with GitHub Actions Error

pothos reviewed Mar 13, 2024

View reviewed changes

changelog/changes/2024-03-13-zfs.md Outdated Show resolved Hide resolved

pothos had a problem deploying to development March 13, 2024 22:23 — with GitHub Actions Error

changelog: Add entry for zfs sysext

3aa9cd2

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>

pothos force-pushed the jepio/zfs-sysext branch from c67c2a4 to 3aa9cd2 Compare March 13, 2024 22:23

pothos had a problem deploying to development March 13, 2024 22:24 — with GitHub Actions Failure

pothos merged commit e586791 into main Mar 13, 2024
1 check failed

pothos deleted the jepio/zfs-sysext branch March 13, 2024 22:24

dongsupark added the main label Mar 14, 2024

github-actions bot mentioned this pull request Mar 22, 2024

Monthly contributions report 2024-02-22 - 2024-03-21 flatcar/Flatcar#1398

Closed

pothos mentioned this pull request Mar 27, 2024

New Package Request: sys-fs/zfs flatcar/Flatcar#1110

Closed

tormath1 mentioned this pull request Apr 26, 2024

Flatcar Podman extension flatcar/Flatcar#112

Open

tormath1 mentioned this pull request Jun 14, 2024

[RFE] Need a good way to add complex Python apps flatcar/Flatcar#1470

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build ZFS sysext with each release #1742

Build ZFS sysext with each release #1742

jepio commented Mar 11, 2024 •

edited

Loading

pothos commented Mar 11, 2024

github-actions bot commented Mar 11, 2024 •

edited

Loading

jepio commented Mar 11, 2024

jepio commented Mar 12, 2024

jepio commented Mar 12, 2024

jepio commented Mar 13, 2024

pothos commented Mar 13, 2024

jepio commented Mar 13, 2024

pothos commented Mar 13, 2024

abuisine commented Mar 22, 2024

jepio commented Mar 22, 2024

abuisine commented Mar 22, 2024

coreyjewett commented Mar 29, 2024

Build ZFS sysext with each release #1742

Build ZFS sysext with each release #1742

Conversation

jepio commented Mar 11, 2024 • edited Loading

Build ZFS sysext with each release

How to use

Testing done

pothos commented Mar 11, 2024

github-actions bot commented Mar 11, 2024 • edited Loading

jepio commented Mar 11, 2024

jepio commented Mar 12, 2024

jepio commented Mar 12, 2024

jepio commented Mar 13, 2024

pothos commented Mar 13, 2024

jepio commented Mar 13, 2024

pothos commented Mar 13, 2024

abuisine commented Mar 22, 2024

jepio commented Mar 22, 2024

abuisine commented Mar 22, 2024

coreyjewett commented Mar 29, 2024

jepio commented Mar 11, 2024 •

edited

Loading

github-actions bot commented Mar 11, 2024 •

edited

Loading