linux: zfs volume snapshot rename does not generate KOBJ_CHANGE, /dev/zvol/.... link is not updated

[JKDingwall]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	Focal / 20.04
Kernel Version	5.4.0-133-generic (local build of Ubuntu tag)
Architecture	amd64
OpenZFS Version	zfs-2.1.6-1 / zfs-kmod-2.1.6-1

Describe the problem you're observing

When a volume snapshot is renamed zfs rename pool/volume@snap pool/volume@other the corresponding /dev/zvol/... link is not updated. Running udevadm monitor shows no uevent is generated for the action. If the volume (not snapshot) is renamed then uevents are generated and all links are correct.

Describe how to reproduce the problem

root@ubuntu:~# zfs create -V 20G ztank/james

root@ubuntu:~# zfs snapshot ztank/james@1

root@ubuntu:~# zfs snapshot ztank/james@2

root@ubuntu:~# ls -l /dev/zvol/ztank/james*
lrwxrwxrwx 1 root root 12 Nov 24 13:00 /dev/zvol/ztank/james -> ../../zd2736
lrwxrwxrwx 1 root root 12 Nov 24 13:00 /dev/zvol/ztank/james@1 -> ../../zd2752
lrwxrwxrwx 1 root root 12 Nov 24 13:00 /dev/zvol/ztank/james@2 -> ../../zd2768

root@ubuntu:~# zfs rename ztank/james@1 ztank/james@3

root@ubuntu:~# ls -l /dev/zvol/ztank/james*
lrwxrwxrwx 1 root root 12 Nov 24 13:00 /dev/zvol/ztank/james -> ../../zd2736
lrwxrwxrwx 1 root root 12 Nov 24 13:00 /dev/zvol/ztank/james@1 -> ../../zd2752
lrwxrwxrwx 1 root root 12 Nov 24 13:00 /dev/zvol/ztank/james@2 -> ../../zd2768

The expected result is that the @1 symlink is removed and an @3 one is created.

Include any warning/errors/backtraces from the system logs

No messages are recorded in syslog or kern.log which indicate an error.

[JKDingwall]

I don't know if this is interesting:

root@ubuntu:~# /lib/udev/zvol_id /dev/zd2736
ztank/james

root@ubuntu:~# /lib/udev/zvol_id /dev/zd2768
ztank/james@2

root@ubuntu:~# /lib/udev/zvol_id /dev/zd2752
Unable to open device file: /dev/zd2752

root@ubuntu:~# ls -l /dev/zd2752
brw-rw---- 1 root disk 230, 2752 Nov 24 13:44 /dev/zd2752

root@ubuntu:~# strace /lib/udev/zvol_id /dev/zd2752
execve("/lib/udev/zvol_id", ["/lib/udev/zvol_id", "/dev/zd2752"], 0x7fffc0e04428 /* 19 vars */) = 0
brk(NULL)                               = 0x56094f044000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffdff3ca380) = -1 EINVAL (Invalid argument)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=38232, ...}) = 0
mmap(NULL, 38232, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f60921ea000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300A\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
pread64(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32, 848) = 32
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\30x\346\264ur\f|Q\226\236i\253-'o"..., 68, 880) = 68
fstat(3, {st_mode=S_IFREG|0755, st_size=2029592, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f60921e8000
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
pread64(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32, 848) = 32
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\30x\346\264ur\f|Q\226\236i\253-'o"..., 68, 880) = 68
mmap(NULL, 2037344, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6091ff6000
mmap(0x7f6092018000, 1540096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7f6092018000
mmap(0x7f6092190000, 319488, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19a000) = 0x7f6092190000
mmap(0x7f60921de000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e7000) = 0x7f60921de000
mmap(0x7f60921e4000, 13920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f60921e4000
close(3)                                = 0
arch_prctl(ARCH_SET_FS, 0x7f60921e9580) = 0
mprotect(0x7f60921de000, 16384, PROT_READ) = 0
mprotect(0x56094ecf9000, 4096, PROT_READ) = 0
mprotect(0x7f6092221000, 4096, PROT_READ) = 0
munmap(0x7f60921ea000, 38232)           = 0
stat("/dev/zd2752", {st_mode=S_IFBLK|0660, st_rdev=makedev(0xe6, 0xac0), ...}) = 0
openat(AT_FDCWD, "/dev/zd2752", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "Unable to open device file: /dev"..., 40Unable to open device file: /dev/zd2752
) = 40
exit_group(1)                           = ?
+++ exited with 1 +++

[ryao]

Thanks for the report. I expect to see this fixed before Christmas.

[JKDingwall]

If there is patch we would be willing to test it.

[ryao]

If there is patch we would be willing to test it.

I had hoped that the group that last touched this infrastructure would write the patch, but they have not. I guess I will need to do it, although I cannot do it immediately.

[JKDingwall]

This is a bit of a sticking point for us at the moment and we've not managed to come up with a workaround. Is it a simple issue that someone unfamiliar with the zfs code base could resolve with a bit of guidance?

[ryao]

Unfortunately, it has been somewhat difficult finding time to do this. I plan to do it, although if you would like it done faster, I suggest contacting Klara Systems to get it done.

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

[SpiffyB]

Has there been any progress on this? This is a serious blocker for a project I'm working on currently.

[JKDingwall]

I can still reproduce this problem in zfs-2.2.6. I've added a couple of printks and for the snapshot rename I can see that zvol_rename_minors_impl() is entered but there is no subsequent call to zvol_os_rename_minor(). I'll continue trying to follow what is happening...

[JKDingwall]

The printk I added to zvol_rename_minor_impl():

        printk(KERN_ERR "ZFS: zvol_rename_minors_impl(%s->%s)...\n", oldname, newname);

If I rename a zvol with snapshots, e.g. zfs rename zpool/james zpool/dingwall:

ZFS: zvol_rename_minors_impl(zpool/james->zpool/dingwall)

If I rename a snapshot , e.g. zfs rename zpool/dingwall@apple zpool/dingwall@orange

ZFS: zvol_rename_minors_impl(apple->orange)...

So I suspect the problem is that zvol_rename_minors_impl() is not called with the full zfs path.

[JKDingwall]

a0bd735 removed code which prefixed fsname to the snapshot name before calling zvol_rename_minors(). That goes back to 0.7.0 (!) but as long time zfs users I can't remember when we first observed this problem. I'll see if I can reinstate the code which was removed.

-#ifdef _KERNEL
-       oldname = kmem_asprintf("%s@%s", fsname, oldsnapname);
-       newname = kmem_asprintf("%s@%s", fsname, newsnapname);
-       zvol_rename_minors(oldname, newname);
-       strfree(newname);
-       strfree(oldname);
-#endif