Problem with upgrade from 0.6.2 (released version)

[FransUrbo]

I noticed this in zfsonlinux/grub#9.

It seems (still not absolutly verified) that there is a problem if/when the module is still 0.6.2, but the userland comes from HEAD. Or at least, that's the only thing I can think of this far.

debian:~# df -h .
Filesystem           Size  Used Avail Use% Mounted on
rpool/ROOT/debian-1   16G  1.2G   15G   8% /
debian:~# zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME                                     STATE     READ WRITE CKSUM
        rpool                                    ONLINE       0     0     0
          ata-VBOX_HARDDISK_VBd6e3d348-7c453f68  ONLINE       0     0     0
          ata-VBOX_HARDDISK_VB49020f73-ef98ed15  ONLINE       0     0     0

errors: No known data errors
[... do the upgrade ...]
debian:~# zpool status
no pools available
debian:~# dmesg | grep ZFS:
[    2.363408] ZFS: Loaded module v0.6.2-525_gd584a01, ZFS pool version 5000, ZFS filesystem version 5
debian:~# dpkg -S `which zpool`
zfsutils: /sbin/zpool
debian:~# dpkg -l zfsutils
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                              Version               Architecture          Description
+++-=================================-=====================-=====================-========================================================================
ii  zfsutils                          0.6.3-0.4~g0ad85e~whe amd64                 command-line tools to manage ZFS filesystems

After a reboot, the pool is back as it's supposed to.

[FransUrbo]

An interesting "sideffect" is that if I boot of a snapshot (which does a clone of the specified snapshot and uses that as root filesystem), a reboot doesn't help!

I'm still able to boot from that resulting dataset so the initramfs can import the pool, see datasets and mount it), but after the system have finished booting, zpool can't find any pools and zfs no datasets... Might be another problem, but worth mentioning anyway.

[behlendorf]

This is absolutely true. Unfortunately, the libzfs_core patvhes we needed to pull in to track Illumos changed the kernel/user ABI. I've talked with upstream about trying to avoid this sort of thing going forward. Apparently this is less of an issue for them since they force kernel/user space to be updated in sync. Obviously it is a big deal for us so post 0.6.2 we've merged a number of changes to help prevent this from being an issue going forward.

Compatibility code could have been written for all of this but we didn't have the resources to devote to it.

[FransUrbo]

So the question is what to do before 0.6.3 is released - document the behavior properly (downside: people don't read documentation - I'm no exception :), just release and deal with all the questions and issue reports about this on an individual basis (in combination with documentation, this seems to be the simplest and easiest thing) or write the comparability code.

Since time is always an issue for everyone, I vote on making this a documentation issue and keep it open for as long as we think we need, write up the "reboot necessary after upgrade" part somewhere safe and easily found and post a link to that in the issue here.

[FransUrbo]

Just a side effect of this problem: When upgrading both ZoL and Grub (because of libzfs SONAME change), the grub.cfg file will be faulty:

        linux   /vmlinuz-3.2.0-4-amd64 root= ro   quiet
                linux   /vmlinuz-3.2.0-4-amd64 root= ro   quiet
                linux   /vmlinuz-3.2.0-4-amd64 root= ro   single

For some people, this will be a nobrainer - just edit the file and set a correct command line. But for those that barely know what an editor is, but still insist on running ZFS... ?

Reinventing the wheel (basically recreating grub-mkconfig and grub-probe etc in shell script just for this purpose) seems to be overkill...

[FransUrbo]

In zfsutils_0.6.3-0.7g0ad85ewheezy_amd64.deb which I just uploaded to the archives, I try to document this as best as I can. A copy of the readme I tell user/admin to read is located at http://bayour.com/misc/README.upgrade_from_0.6.2.

It probably needs work, but it might be a start.

[behlendorf]

@FransUrbo My intention was to do the following:

• Document this clearly in the release notes. Most people won't read but some will and it will provide something we can reference when people have an issue.
• Mitigate the grub issue as much as possible by providing a libzfs.so.1 library if at all possible. The functionality grub requires is minimal and as you already know the fact that a symlink works pretty much verifies the ABI hasn't changed.
• Make sure any additional compatibility breakage which we know is going to happen and can't be avoided gets merged now before the 0.6.3 tag is made. Also make sure we can minimize issues going forward, see changes like 2e0358c. It's better that we only go through this once if possible.

[FransUrbo]

Document this clearly in the release notes.

Also a little note on the front page (next to the 'Changelog' entry) would be nice. Something like a little asterisk and a 'read the following page if you're upgrading from 0.6.2 or earlier'. People should really read the Changelog before upgrading, but there you go... :)

What I mean is, that this upgrade is quite disruptive - more so than other's I've seen. We really need people to pay attention to this upgrade, and an extra note somewhere next to the coming '0.6.3 release line' would be prudent.

People is going to miss that (or ignore it any way), but the more places we put it in, the bigger the chance they'll read it.

Mitigate the grub issue as much as possible by providing a libzfs.so.1 library if at all possible.

That might not be necessary - grub-probe will fail even if there IS such a link. It fails because it can't retrieve the pool information (because the pool have 'vanished").

The link is only necessary if it's not recompiled against the new libzfs2. And I have already supplied such a recompiled version, and I'm sure @dajhorn will do to when it's time for release (if he haven't already - he's been doing snapshots/dailies longer than me :)...

Just documenting that "after the upgrade, either a recompilation of grub OR creating a symlink will be required" would probably be enough.

Or even better: Saving the running grub.cfg in a safe place before starting the upgrade and then copying it back is best.

I can't do that in the packages ("I" don't own it, so I'm not allowed to touch it - also, I can't be sure it haven't been modified wrongly already)..

So unless you can write a 'mini-libzfs1' library that will allow the pool to be viewed ("zpool status" etc) with newer binaries, older module, documenting the problem as best as we can is the only way forward.

[wom-bat]

I'm seeing the same issue after upgrading to 0.6.4

I see after boot:

 SPL: Loaded module v0.6.4-1b-wheezy
 ZFS: Loaded module v0.6.4-1.1-Z-wheezy, ZFS pool version 5000, ZFS filesystem version 5

Command:
Message:
Error:

Manually import the root pool at the command prompt and then exit.
Hint: Try: zpool import -R / -M
....
# zpool import -R / -N
# zpool status
no pools available
# zdb | head
bucket:
    version: 5000
    name: 'bucket'
    state: 0
    txg: 11288530

cmdline is:
BOOT_IMAGE=/vmlinuz-3.16.0-4-amd64 bootfs=bucket/ROOT/lemma-1 ro boot=zfs rootdelay=10

Help?