Boot environment contents: /etc, /usr and /var filesystems

[jasonbstark]

Thanks to Sithuk for the excellent Ubuntu-Server-ZFSbootmenu scripts.

I have a question. According to the ZFSBootMenu Boot Environments Primer:

ZFSBootMenu is certainly capable of booting an environment that mounts separate filesystems at / and other paths like /etc, /usr or /var. ... However, a key benefit of boot environments is atomicity. In general, it is bad to allow the contents of /usr to become inconsistent with the package manager database on /var. Configuration files in /etc can often be tied to specific versions of software, so they should be kept consistent as well. When these directories live on different filesystems, ensuring consistency becomes much more challenging.

This seems to argue for the contents of /etc, /usr and /var to be included as datasets under rpool/ROOT/ubuntu.2023.12.04, rather than separate, independent datasets.

What is the rationale for creating /usr, for instance, using zfs create -o canmount=off rpool/usr, instead of using zfs create -o canmount=off rpool/ROOT/ubuntu.2023.12.04/usr?

If /usr were created using the latter method, snapshots of the entire boot environment could be taken using zfs snapshot -r rpool/ROOT/ubuntu.2023.12.04@snap_name. This would ensure that all boot environment contents would be contained in snapshots that were taken as simultaneously as possible.

Is there a problem created by using the latter method?

[Sithuk]

The canmount=off property doesn't make a separate mountable filesystem. It is to establish inheritable properties for descendent file systems. So /usr and /var remain coupled with the system state.

You can test this for yourself by cloning a snapshot a few times to create test boot environments (BE). Then create a test file in /usr and /var on each boot environment. Name them differently in each BE to allow for comparison. Then switch between the boot environments to check that the system state remains consistent. You can also test /var/lib/apt and /var/lib/dpkg if you have specific concerns about losing package manager consistency with the system state.

Relevant man page entry:
https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html#canmount

If this property is set to off, the file system cannot be mounted, and is ignored by zfs mount -a. Setting this property to off is similar to setting the mountpoint property to none, except that the dataset still has a normal mountpoint property, which can be inherited. Setting this property to off allows datasets to be used solely as a mechanism to inherit properties. One example of setting canmount=off is to have two datasets with the same mountpoint, so that the children of both datasets appear in the same directory, but might have different inherited characteristics.
When set to noauto, a dataset can only be mounted and unmounted explicitly. The dataset is not mounted automatically when the dataset is created or imported, nor is it mounted by the zfs mount -a command or unmounted by the zfs unmount -a command.

This property is not inherited.

The reason to create rpool/usr is to allow the creation of a persistent rpool/usr/local so that locally compiled software is persistent across boot environments. The approach is as per the OpenZFS root on zfs guide for Debian, which doesn't use zsys. Link below.
https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Bookworm%20Root%20on%20ZFS.html#step-3-system-installation

[jasonbstark]

Thank you, Sithuk. I see. May I ask you to please correct me, if I'm wrong? After booting from the Live ubuntu boot device, pool $RPOOL"/ROOT/"$rootzfs_full_name is created with canmount=noauto and mountpoint=/. Other datasets, such as "$RPOOL"/var and "$RPOOL"/var/lib are created with canmount=off. Then, debootstrap "$ubuntuver" "$mountpoint" installs linux into mountpoint, creating and populating directories like /var/lib/apt.

There is no dataset that mounts at /var/lib/apt, so snapshots of $RPOOL"/ROOT/"$rootzfs_full_name and its children will contain /var/lib/apt. This will hold them in any boot environment that results from recursively cloning $RPOOL"/ROOT/"$rootzfs_full_name.

Dataset "$RPOOL"/var/snap is created and mounted at its default location (/var/snap) under the mountpoint for "$RPOOL", to explicitly create that location prior to installing linux. All snap contents are then excluded from zfs recursive snapshots of $RPOOL"/ROOT/"$rootzfs_full_name.

Dataset "$RPOOL"/var/lib is created and mounted, so that "$RPOOL"/var/lib/docker can be created and excluded from zfs snapshots of $RPOOL"/ROOT/"$rootzfs_full_name.

If I create a dataset "$RPOOL"/var/nc_data under "$RPOOL"/var, and mount it at /var/nc_data, then all of its contents will be held in snapshots of "$RPOOL"/var/nc_data. This would all be independent of the snapshots that are cloned from the boot environment.

Thank you for your help, and for providing this excellent script!

[jasonbstark]

And, then, in using ZFSBootMenu, new boot environments would be created by cloning snapshots of $RPOOL"/ROOT/"$rootzfs_full_name.

[Sithuk]

Dataset "$RPOOL"/var/lib is created and mounted

/var/lib is created with canmount=off. So it won't be separately mounted.

If I create a dataset "$RPOOL"/var/nc_data under "$RPOOL"/var, and mount it at /var/nc_data, then all of its contents will be held in snapshots of "$RPOOL"/var/nc_data. This would all be independent of the snapshots that are cloned from the boot environment.

The script is currently setup to take snapshots of "$RPOOL"/ROOT. So your /var/nc_data would not be automatically snapshotted. You would need to configure that.

And, then, in using ZFSBootMenu, new boot environments would be created by cloning snapshots of $RPOOL"/ROOT/"$rootzfs_full_name.
Yes, that's correct.

I recommend testing your setup in a VM first so you can confirm the behaviour is as you expect. Please let me know how you get on.

[jasonbstark]

Thank you, Sithuk, for your reply. Yes, it is clear now. I appreciate your help. I have been using zrepl for scheduled snapshots, which also sends backups over TLS to a zfs remote server. It keeps the snapshots pruned, with separate schedules, on server and remote.

I will definitely work it through in a VM before taking it live. Will keep you posted! Thanks!

[jasonbstark]

I created a VirtualBox VM, with bridged network, and successfully installed ZFSBootMenu, using your script. It took several attempts, due to the VM aborting, typically when writing initramfs. I repeated the install until no aborts occurred.

I was able to enter the ZBM facility, and to create snapshots and alternate boot environments. This all seems to work properly. ZBM is really going to be an asset, when it comes to doing system upgrades. Your script is a huge asset!

Before going live, I want to work out the remote access on my VM. I have tried a couple of times, but have not succeeded. I have confirmed that the VM can ping my laptop, and that my laptop can ping the VM. However, I cannot ssh from laptop to VM, in order to copy my ssh public key.

Would you be able to provide any guidance? I have attached the standard output from your remote access script.

Thank you for your help!

remote_ZBM.out.2023.12.15_12.30.10.txt

[Sithuk]

I can see from your txt file that the following was correctly printed at the end of the remote access install. Did you install openssh-server to allow you to ssh in as the message says? Setting the "extra_programs" variable to "yes" would install openssh-server when running the script during the initial system installation.

Zfsbootmenu remote access installed. Connect as root on port 222 during boot: "ssh root@{IP_ADDRESS or FQDN of zfsbootmenu} -p 222"
Your SSH public key must be placed in "/home/jasonstark/.ssh/authorized_keys" prior to reboot or remote access will not work.
You can add your remote user key using the following command from the remote user's terminal if openssh-server is active on the host.
"ssh-copy-id -i ~/.ssh/id_rsa.pub jasonstark@{IP_ADDRESS or FQDN of the server}"
Run "sudo generate-zbm" after copying across the remote user's public ssh key into the authorized_keys file.

[jasonbstark]

Yes, that was the problem. I installed openssh-server, along with the other extras in the extra_programs list. I confirmed that the VM was accessible using ssh. I copied in my public key, and used your script to execute the remote_access option.

Everything is now working perfectly. I can reach the zfsbootmenu prompt, and provide the encryption key remotely. It's a beautiful thing!

Thank you!

[Sithuk]

Thank you for the feedback. I'm glad you got everything working.