snapshotting top-level "bpool" filesystem causes grub to fail #13873
Comments
|
This looks like a bug in grub. Let us keep this open to track the issue (and encourage patches), but someone hound file a bug with the GRUB project. |
|
I still encounter this error running Arch with Grub version 2:2.12rc1-5. I had to destroy the pool and recreate one, then disable snapshotting on the "boot" pool as a temporary workaround. Alternatively, one can take snapshots of the datasets but not the pool. Edit: |
|
I'm using grub grub-probe (GRUB) 2.12~rc1-12 (But I looked at grub master, without notable difference) I got this problem after updating from zfs 2.1.4 to 2.2.2 (without zfs upgrade). a grub-install -v -v says: Following lines seem to bail out: The thing is: "inherit" is not a compression type per se, but says "use whatever its parent (or parent-parent or ...) is using). So, it has no decompress method: static decomp_entry_t decomp_table[ZIO_COMPRESS_FUNCTIONS] = { I think the missing piece that grub does not look up the parents if inherit. |
|
I can confirm that using grub 2.12 does indeed help. |
How does one upgrade grub in Ubuntu 22.04 LTS in particular when I can't boot at all? I don't understand how it broke all of a sudden, I've been taking snapshots for years (with sanoid). |
You can e.g. start a live version, install zfs drivers and import bpool. Then you can make an USB stick with grub and copy your kernel + initrd + the grub config needed to start this on it. Then you can boot your installation with it (don't forget to export bpool before ;) Another way is to import both bpool and rpool, mount it together, bindmount dev, run, sys and proc and chroot it. I used this way on debian bookworm. Either way, you can either try to install grub 2.12 packages from newer distro versions, or build grub manually (it isn't too complicated, just ensure that you have the zfs and devicemapper libs installed,the configure script will tell you if you have...) and, if you use uefi boot, use the --with-platform=efi . Then, a grub-install does the job and you have (hopefully) a bootable system again. |
|
I managed to boot it for now using portable ZfsBootMenu running from a flash drive. But still unclear what I should do to fix Ubuntu 22.04. I don't have a bpool, this is an old setup from the 18.04 days, there's a single rpool that contains |
|
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/2041739/comments/9 zpool create \
-o feature@extensible_dataset=disabled \
-o feature@bookmarks=disabled \
-o feature@filesystem_limits=disabled \
-o feature@large_blocks=disabled \
-o feature@large_dnode=disabled \
-o feature@sha512=disabled \
-o feature@skein=disabled \
-o feature@edonr=disabled \
-o feature@userobj_accounting=disabled \
-o feature@encryption=disabled \
-o feature@project_quota=disabled \
-o feature@obsolete_counts=disabled \
-o feature@bookmark_v2=disabled \
-o feature@redaction_bookmarks=disabled \
-o feature@redacted_datasets=disabled \
-o feature@bookmark_written=disabled \
-o feature@livelist=disabled \
-o feature@zstd_compress=disabled \
-o feature@zilsaxattr=disabled \
-o feature@head_errlog=disabled \
-o feature@blake3=disabled \
-o feature@vdev_zaps_v2=disabled \
[...]
|
|
FWIW, I fixed my issues by moving to ZFSBootMenu and couldn't be happier. Excellent piece of software to pair with ZFS! |
|
From what I can tell, this bug may have been around for up to 7 yrs. I resolved the problem on my Ubuntu Mantic system by deleting the bpool and recreating with following zfs options: Mathias Aerts identified that the culperate zfs flag is feature@extensible_dataset and has had success using a bpool created with all extra features disabled. Maybe update-grub/grub-probe should check the zfs version/flags/features before trying to do anything? |
|
this is correct ^ |
|
The solution with disabling all features (except those not supported by the kernel) listed in @n0099 post works for me so far. I advise against just installing grub 2.12 to your boot environment unless you also update grub packages in your system to 2.12. Old grub tools (<2.12) in your system won't be able to detect fs_uuid of bpool and won't properly generate menu entries. This will bork your boot menu once again the next time there's a new kernel version when entries are regenerated. |
|
creating a new bpool isnt a solution, it is a workaround, and a very poor one at that. seeing how this isnt actually a grub issue, it should likely be receiving more attention @pcd1193182 |
|
@bghira : Hmm? It is actually a grub issue. Between grub 2.12 rc1 and grub 2.12 release there were 4 bug fixes ( https://git.savannah.gnu.org/cgit/grub.git/log/?h=grub-2.12&qt=grep&q=zfs , I mean the 4 from 2023-09-18). It is left for someone else as exercise to bisect which one is the bug fix for the problem here. Probably grub traps over one of these bugs when the pool gets used (completely legal) in certain ways. Yes, installing 2.12 in the boot environment is a one time hack. But e.g. debian sid has 2.12 , which can be installed in bookworm as well without much hassle. I did exactly this in my case ;-) |
Ok so you destroyed the bpool and created it with the parameters. |
which OS are you on (i.e. arch, ubuntu, fedora, etc)? The general idea is to create the zfs bpool, chroot into the OS (from an external USB drive), install a bootloader (usually grub, but could be something else), re-generate initrmfs, probably need to re-install kernels (and microcode) as well. I know how to do it on Arch, so let me know if you're on Arch or Arch based OS, i can write you the exact steps. cheers. |
Many thanks! I tried steps as described here, but already fail at the first command of Step 5 with Translation: |
|
That guide was ambiguous when a command is run as root (#) or a regular user ($), in this case, If you're under a non-root user, please try: Or login as root: |
|
Many thanks! I have followed this guide and it worked. I do not want to break it right now. |
|
After installing noble's 2.12 packages into a mantic install, grub-install and update-grub, and snapshotting bpool, my boot environment is now broken. I would respectfully suggest that a 2.12 install may not be the fix that we are hoping it may be. I shall recreate with the options @dannyp777 suggests, and disable extensible_dataset in addition. |
I took a copy of /boot before I deleted the bpool then just copied it back once I had recrecreated the bpool. Sorry, I hadn't meant my original reply to be a comprehensive how-to. There are more details of things people tried over at the launchpad bug report. |
|
Using ZFSBootMenu is another, much easier option. |
|
HI, I was just struggling with this issue on Debian bookworm (grub-efi (2.06-13+deb12u1)). I've recreated my boot pool with Then I've recreated the pool again and installed manually grub-efi from backports: which installed 2.12-1~bpo12+1 version of the package (and updated dependencies appropriatelly). My OS survived two restarts, one before snapshot and another after one snapshot. Still testing and observing... |
Interesting, I am getting this message when trying to install "grub-efi/bookworm-backports: Deepl translates this to: |
|
@SimonBard Do you have appropriate apt sources in your apt config, as explained here? |
Sorry, my bad. I did not associate bookworm with debian. I am using ubuntu 22.04 Should I install grub from live-dvd? |
|
@SimonBard I haven't tried it by myself, but it looks like using backports on Ubuntu is pretty simillar to using backports on Debian. https://help.ubuntu.com/community/UbuntuBackports Sad news is that probably grub-efi is missing from jammy-backports https://packages.ubuntu.com/search?suite=jammy-backports&keywords=grub-efi |
Could you probably explain?
There is no pool without the toplevel filesystem ,so I do not understand i! |
|
please ignore the sanoid part, i just did another experiment, and it was not accurate, so i just crossed it out from the original comment. About taking the snapshots of datasets vs pool, i meant this: Let's say you have A a temp workaround, please try only taking snapshots of |
Your answer is incomplete and will only create more issues. For some reason, grub2 doesn't seem to like the idea of a root dataset. The grub shell won't be able to list any files there. In order to mimic whatever Ubuntu installer did when the system was installed, you need to do something like this: and then create filesystem dataset to act as a container [1]: and create filesystem dataset [1]: only then you can mount it at an alternate path and restore all the files and directories from step you mentioned above. Once done, you can verify it from within chroot environment with: It should list zfs as the filesystem. As a last thing, you need to tweak the new bpool mountpoint by changing existing or re-creating the zfs-list.cache file (you already have one pointing to old mount point) [2]: |
I think it was ZFS: Check bonustype in addition to dnode type. |
|
confirming grub 2.12 works here too! I had an EFI loader created a year ago with one of the previous gurb versions. Yesterday I made a snapshot and ran in this issue. Creating and installing new EFI loader with the 2.12 solved the problem. |
|
I recently tried to boot old (10.3) FreeBSD from a pool restored from backup on newer version, and it failed despite all disabled features until I've redone it on the same old FreeBSD version. I wonder if we leak some incompatible changes despite disabled features. |
|
This seems to be fixed at least with debian bookworm-backports grub-efi 2.12 packages. |
GRUB is not able to detect ZFS pool if snaphsot of top level boot pool is created. This issue is observed with GRUB versions up to v2.06 if extensible_dataset feature is enabled on ZFS boot pool. compatibility=grub2-2.06 would enable all read-only compatible zpool features except extensible_dataset and other features that depend on it. The existing grub2 compatibility file is now renamed to grub2-2.12 to reflect the appropriate grub2 version. grub2-2.12 lists all read-only features that can be enabled on boot pool for grub2 with version 2.12 onwards. A new symlink grub2 is created that currently points to the grub2-2.12 compatibility file. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Alexander Motin <mav@FreeBSD.org> Signed-off-by: Umer Saleem <usaleem@ixsystems.com> Closes openzfs#13873 Closes openzfs#15261 Closes openzfs#15909
System information
Describe the problem you're observing
I've noticed last week that my system stopped being able to find my zpool which has disabled features so that Grub can detect it.
I ran
grub-probe /path/to/bpooland it shows the error:Tried setting compression algorithm explicitly on each filesystem in the bpool, no change.
Describe how to reproduce the problem
I recreated the pool:
And then, the command works:
I then snapshot the child filesystem:
I can snapshot the lowest level child filesystem, and I then snapshot the pool itself:
That's when things go south. Up until this point, I can reboot readily and Grub works just fine. This was working for a very long time, and I haven't upgraded Grub at all.
The text was updated successfully, but these errors were encountered: