New Package Request: sys-fs/zfs

[jhaprins]

Package name and purpose
A filesystem perfectly suited for hosting the storage in a Kubernetes environment.

• Package upstream repo: https://github.com/openzfs/zfs
• Gentoo ebuild: https://packages.gentoo.org/packages/sys-fs/zfs

Impact of adding this package to the Flatcar OS image

The package improves on the following core values:

• [ ? ] Secure by default
• [ ? ] Always up to date
• [ x ] Improve container experience
• [ x ] Operate at scale / automation / telemetry

The package will increase the image size by: Unknown MBytes.

How might this package increase the attack surface:

• [ e.g. new ports open, new services running, etc. ]

Benefits of adding this package
This filesystem is promoted to be the best for storage in a K8S environment. I think I also saw talks and documentation from the Flatcar community that said this filesystem should be used when available. This was the reason for us when we did the initial selection to mark that this was in this OS. Finding this is not the case, surprised us a little, and now I'm investigating what needs to be done to get it in the OS.
I don't know if adding this package is the only thing needed to get this working, I can imagine that some kernel modules are also needed to get all ZFS functionality.
A side note, I don't need to be able to boot from a ZFS filesystem or anything like that. I just need to be able to create ZFS storage for the K8S cluster that is hosted on the Flatcar systems.

Additional information
[ Please add any information that does not fit into any of the above sections here ]

[donch]

Hi, this module and the ZFS tools are essential in a Kubernetes environment to manage persistent storage.
I see that you are likely adding the module to Flatcar according to the roadmap which is great.

For now, we've forked the repo systext-bakery and built the ZFS module for our use. It's currently a proof of concept, but we'll use it in our production environment soon.

This might help the integration of the ZFS module in Flatcar.

[pothos]

Interesting, since the image is pretty much linked to the base OS version I think you should use VERSION_ID= to ensure that it only gets loaded when compatible.
By the way, there is also a helper script for emerge sysext images, started from within the the Flatcar SDK Docker image (which itself is started with ./run_sdk_container): https://github.com/flatcar/scripts/blob/main/build_sysext
It assumes you have checked out the release branch/tag, added your ebuilds on top, and also have downloaded the production image.

[jhaprins]

Very nice.
I'm currently building my own image for the K8S environment we are going to build, and this is mainly because there are more things that we need. One of the things being snmpd to be able to monitor the nodes in the cluster.
The version I'm currently building is the latest Alpha of Flatcar, which is the only version that has the sysext scripting included in the buildtree.
What do I need to do to get ZFS working in our images with your sysext-bakery clone / how to I integrate your clone with the build repository for the latest Alpha of Flatcar?

[donch]

@pothos , in fact we re-use this work : https://github.com/jepio/flatcar-zfs-overlay/blob/main/Dockerfile to build the extension. But you are right, the extension is linked to a specific Flatcar version (since we build the modules with a specific kernel version). Also, as we build the module ourselves, it's not signed with the key used to build the kernel, so the kernel is taint, but i'm not sure it's a issue.
Regarding the VERSION_ID, i'll have a look, thanks.

@jhaprins , you can clone the repo and modify somes files to build the right module version.
In the github action, you should define in the matrix part, the exact version of flarcar you want to build the module against.
And also you should duplicate the directory 3510.2.6 (https://github.com/enix/sysext-bakery/tree/main/files/zfs) for your Flarcar version. If your Flatcar vers has a newer kernel version, you may want to copy the ebuild of ZFS 2.1.12 (from the gentoo repository which require python 3.10).

[coreyjewett]

@donch have you worked out the secret sauce for systemd to load the zfs module and import your pool before docker starts? That's where I'm stuck. My home fileserver was happily running zfs using torcx, but trying to migrate to flatcar has been painful. Doubly so as I don't have much time to poke at it unfortunately.

I forked jepio's repo too, but added github workflows that build and publish both a docker image and squashfs raw file. It also includes the VERSION_ID to pin the sysext to the Flatcar release. The following snippet of butane works happily to get the sysext:

storage:
  files:
      - path: /etc/extensions/flatcar-zfs-overlay-${FLATCAR_VERSION}.raw
      mode: 0644
      contents:
        source: https://github.com/coreyjewett/flatcar-zfs-overlay/releases/download/flatcar-${FLATCAR_VERSION}/flatcar-zfs-overlay-${FLATCAR_VERSION}.raw

(I have a script that subs in the version ahead of transpiling to ignition.)

So I successfully have the overlay fully loaded, but it isn't automatically loading the kernel module. I can manually modprobe zfs, but I really want to have my pool loaded before docker starts since all my volume mounts live there. (not using k8s, too much overhead for my purposes)

[jepio]

You'll have to make systemd-modules-load.service run after the sysext service (via dropins), and enable all the zfs services/targets manually. And then you'll probably also need to add a dependency between containerd (or docker) and zfs.target (I think).

I wrote about this problem here: https://groups.google.com/g/flatcar-linux-user/c/RX9ZO_hX1JA/m/OLgMYoTHAgAJ.

[donch]

Exact.
In our case, we use 2 dropin : https://github.com/enix/sysext-bakery/tree/main/files/zfs/usr/lib/systemd/system

• 1 after sysext loading to reload modules
• 1 for the zfs pool import (which is done after the modules loading)

[jhaprins]

ebuild of ZFS 2.1.12

* ERROR: sys-fs/zfs-kmod-2.1.12::zfs-overlay failed (depend phase):
 *   linux-mod-r1.eclass could not be found by inherit()
 * 
 * Call stack:
 *                ebuild.sh, line 628:  Called source '/var/lib/portage/zfs-overlay/sys-fs/zfs-kmod/zfs-kmod-2.1.12.ebuild'
 *   zfs-kmod-2.1.12.ebuild, line   6:  Called inherit 'autotools' 'dist-kernel-utils' 'flag-o-matic' 'linux-mod-r1' 'multiprocessing'
 *                ebuild.sh, line 273:  Called die
 * The specific snippet of code:
 *   		[[ -z ${location} ]] && die "${1}.eclass could not be found by inherit()"
 * 
 * If you need support, post the output of `emerge --info '=sys-fs/zfs-kmod-2.1.12::zfs-overlay'`,
 * the complete build log and the output of `emerge -pqv '=sys-fs/zfs-kmod-2.1.12::zfs-overlay'`.
 * Working directory: '/usr/lib/python3.11/site-packages'
 * S: '/var/tmp/portage/sys-fs/zfs-kmod-2.1.12/work/zfs-kmod-2.1.12'
 * ERROR: sys-fs/zfs-kmod-2.1.11-r1::zfs-overlay failed (depend phase):
 *   linux-mod-r1.eclass could not be found by inherit()
 * 
 * Call stack:
 *                   ebuild.sh, line 628:  Called source '/var/lib/portage/zfs-overlay/sys-fs/zfs-kmod/zfs-kmod-2.1.11-r1.ebuild'
 *   zfs-kmod-2.1.11-r1.ebuild, line   6:  Called inherit 'autotools' 'dist-kernel-utils' 'flag-o-matic' 'linux-mod-r1' 'multiprocessing'
 *                   ebuild.sh, line 2[73](https://github.com/jhaprins/sysext-bakery/actions/runs/6082233613/job/16499561056#step:4:73):  Called die
 * The specific snippet of code:
 *   		[[ -z ${location} ]] && die "${1}.eclass could not be found by inherit()"
 * 
 * If you need support, post the output of `emerge --info '=sys-fs/zfs-kmod-2.1.11-r1::zfs-overlay'`,
 * the complete build log and the output of `emerge -pqv '=sys-fs/zfs-kmod-2.1.11-r1::zfs-overlay'`.
 * Working directory: '/usr/lib/python3.11/site-packages'
 * S: '/var/tmp/portage/sys-fs/zfs-kmod-2.1.11-r1/work/zfs-kmod-2.1.11'

[jhaprins]

I was able to build your overlay module for version 3689 with ZFS 2.1.11. Now lets see if it loads.

[krishjainx]

@jhaprins Could you please provide an update on this ticket? Thank you

[jhaprins]

The module loads and I can see the kernel modules loaded and everything, so that looks all fine. Have not yet tested creating filesystems etc, because the test system I'm currently using it on does not have extra disks installed. Will try to get this done somewhere later this week, can't promise at the moment though.

[jhaprins]

@donch do you have an explanation why a test pool I created won't import on boot?
I see the following in the logfiles, which might be related:
Sep 18 08:50:19 node-1 systemd[1]: zfs-import-scan.service - Import ZFS pools by device scanning was skipped because of an unmet condition check (ConditionFileNotEmpty=!/etc/zfs/zpool.cache).

[donch]

Hi @jhaprins , i had this error when i upgraded Flatcar and ZFS version. The pool scanning wasn't ran because of systemd condition. In my case i had to remove the file /etc/zfs/zpool.cache after the upgrade, then the pool is correctly discover at the next boot.

[jhaprins]

@donch Could you contact me regarding your sysext-bakery repository? I'm running into some issues when I try to use your repo so I can build against the latest beta of Flatcar. I would be happy to do this outside of this ticket. You can mail me at jhp at jhprins dot org

[jhaprins]

@donch I'm currently working on building the sysext module in the SDK environment instead of the sysext-bakery, that way it will be easier to integrate it later into the central build repository of Flatcar itself. In your sysext-bakery repo you build everything and in the image the files belonging in /etc are located in /usr/etc. Where in the whole load process do you put them into /etc? I can't seem to find this.

[pothos]

A Flatcar ZFS extension is released as part of the latest Alpha (flatcar/scripts#1742)