Installs on ZFS don't use ZFS snapshotter

[apnar]

Currently if microk8s is installed on a ZFS filesystem it will use the "native" containerd snapshotter instead of the "zfs" snapshotter. That allows it to function but is not ideal or efficient. With ZFS gaining support in Ubuntu, including the ability to install root on ZFS in 20.04 it'd be better to detect and handle this properly.

A few thoughts/ideas:

• leave the current behavior but create a zfs-snapshotter add-on that will interactively enable the feature allowing the user to select a pool
• detect ZFS on install and ask a few interactive questions during installation
• detect the standard Ubuntu root install ZFS layout and automatically create a dataset in that rpool at installation
• detect the zfs pool that /var/snap/microk8s lives in and automatically create a dataset in that pool at installation (the disk space would have come out of this pool anyway with the default behavior)

A similar approach could be used for the root storage pool.

Related previous activity:
Issue #401 and PR #1000

[ktsakalozos]

This is interesting and look like a good fit for an enable addon as it requires user interaction. Some thoughts/questions:

• how would the interface look like?
• we should expect the admin to have pools already created and just detect and point to the. We may offer some instructions on how to create pools.
• note there is no way to interact with the user when snap install microk8s
• we have to investigate if we will need a special interface in a strictly confined setup
• how would the pool detection be done? Will we need to package any binaries?

[dalbani]

MicroK8s uses the native snapshotter in case of a ZFS filesystem since #1000.
What's the advantage of using the zfs snapshotter then? There's not much information on https://github.com/containerd/zfs.

[ktsakalozos]

@dalbani, this a question for the containerd project.

I can only tell you that our decision of going with the "native" snapshotter is based on how less intrusive it is, ie we would not need to create a zfs dataset as described in #401 (comment)

[myers]

Having used the zfs snapshotter for the past 2 years I'd like to throw my $0.02 in. DON'T. The current verison of ZFS tools (like the zfs cli) are not ready for how the zfs snapshotter uses datasets. Docker images are made up of layers, 10's up to 100's of layers per image. Each of the these layers get created as a dataset in zfs. On my system there are over 20,000 of these datasets. zfs list takes over a minute to run. update-grub takes 6 minutes to run because it does zfs list once for every pool in your system (probably something they can improve).

I suspect the snapshotter maybe loosing track of these datasets, as I think 20,000 seems a bit high.

I am now running microk8s on a zvol. That's a part of a zfs pool that you can format as any filesystem. The big downside is that it's a fixed size.

how to run microk8s on zfs using a zvol

The pool we want to store the microk8s image files and container fs is rpool. We want to make a 900GB volume. This will make you download all the images for your pods again.

As root do:

microk8s.stop
mv /var/snap/microk8s/common/var/lib/containerd /var/snap/microk8s/common/var/lib/containerd.backup
mkdir /var/snap/microk8s/common/var/lib/containerd
zfs create -V 900gb rpool/containerd
mkfs.ext4 /dev/zvol/rpool/containerd
echo "/dev/zvol/rpool/containerd /var/snap/microk8s/common/var/lib/containerd ext4 errors=remount-ro 0 2" >> /etc/fstab
mount /var/snap/microk8s/common/var/lib/containerd
microk8s.start

[dalbani]

Thanks @myers for your feedback. Although running MicroK8s on ZFS, I'll skip the ZFS snapshotter for now.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dalbani]

Unstale.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.