Snapshot directories (.zfs) not created and/or inaccessible

[snhrdt]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	18.04 LTS
Linux Kernel	4.15.0-43-generic
Architecture	x86_64
ZFS Version	0.7.5-1ubuntu16.4
SPL Version	0.7.5-1ubuntu1

Describe the problem you're observing

After using

$ zfs snapshot -r z@example

not all (sub)filesystems have a .zfs directory which enables easy access to snapshots without needing to manually mount them.

NOTE: I understand that the .zfs directories are hidden by default, and that this behavior can be configured via zfs set snapdir=visible z - but this is not the issue here. The .zfs folder is not there, and it cannot be directly accessed by doing ls /z/.zfs/snapshot or using cd to get into it.

Describe how to reproduce the problem

Create ZFS pool, create some filesystems, then create a recursive snapshot and try to access the .zfs snapshot directory on each of the filesystems; it fails on some of them while working on others.

$ zfs create z/influxdb
$ zfs create z/mysql
...
$ zfs snapshot -r z@example
$ ls -l /z/influxdb/.zfs/snapshot
ls: cannot access '/z/influxdb/.zfs/snapshot/': No such file or directory
$ ls -l /z/mysql/.zfs/snapshot
total 0
drwxrwxrwx 2 root root 2 Feb 10 16:36 shares
drwxrwxrwx 2 root root 2 Feb 10 16:09 snapshot

Include any warning/errors/backtraces from the system logs

There are no explicit errors in the log, but I see an awful lot of the following log lines:

Feb 10 16:36:46 node-0 zed: eid=505374 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:46 node-0 zed: eid=505375 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:46 node-0 zed: eid=505376 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:46 node-0 zed: eid=505377 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:53 node-0 zed: eid=505378 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:53 node-0 zed: eid=505379 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:53 node-0 zed: eid=505380 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:53 node-0 zed[7208]: Missed 1 events
Feb 10 16:36:53 node-0 zed: eid=505395 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:53 node-0 zed[7208]: Missed 1 events
Feb 10 16:36:53 node-0 zed: eid=505443 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:53 node-0 zed[7208]: Missed 1 events
Feb 10 16:36:53 node-0 zed: eid=505489 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:53 node-0 zed[7208]: Missed 1 events
Feb 10 16:36:53 node-0 zed: eid=505523 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:53 node-0 zed[7208]: Missed 1 events
Feb 10 16:36:53 node-0 zed: eid=505570 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:53 node-0 zed[7208]: Missed 1 events
Feb 10 16:36:53 node-0 zed: eid=505622 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:53 node-0 zed[7208]: Missed 1 events
Feb 10 16:36:53 node-0 zed: eid=505669 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:54 node-0 zed[7208]: Missed 1 events
Feb 10 16:36:54 node-0 zed: eid=505717 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:54 node-0 zed[7208]: Missed 1 events
Feb 10 16:36:54 node-0 zed: eid=505758 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:54 node-0 zed: eid=505759 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:54 node-0 zed: eid=505760 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:54 node-0 zed: eid=505761 class=history_event pool_guid=0xA997F68F1432377E
Feb 10 16:36:54 node-0 zed: eid=505762 class=history_event pool_guid=0xA997F68F1432377E

[snhrdt]

It looks like commit f1a05fa solved a very similar issue reported in #1141 - but I encounter the same issue with a recent version of ZFS, while the commit is from 2012. Maybe there is some sort of regression here?

[loli10K]

I am not able to reproduce on clean Ubuntu18 install:

root@ubuntu:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.1 LTS
Release:	18.04
Codename:	bionic
root@ubuntu:~# uname -r
4.15.0-45-generic
root@ubuntu:~# zfs create z/influxdb
root@ubuntu:~# zfs create z/mysql
root@ubuntu:~# zfs snapshot -r z@example
root@ubuntu:~# ls -l /z/influxdb/.zfs/snapshot
total 0
drwxrwxrwx 1 root root 0 Feb 10 17:33 example
root@ubuntu:~# ls -l /z/mysql/.zfs/snapshot
total 0
drwxrwxrwx 1 root root 0 Feb 10 17:33 example
root@ubuntu:~# 

One thing to note: there is something wrong with the output provided in the report, the last command suggest the presence of directories shares/snapshot inside ".zfs/snapshot/":

$ ls -l /z/mysql/.zfs/snapshot
total 0
drwxrwxrwx 2 root root 2 Feb 10 16:36 shares
drwxrwxrwx 2 root root 2 Feb 10 16:09 snapshot

[devZer0]

i have made the experience that it can make quite a difference to access a path inside .zfs/subdir before accessing/cd'ing .zfs on the top level. it must have to do something with automount, as you can see when you cross the .zfs dir and access snapshots , some new mount appears for every snasphot

[MirisWisdom]

Had the exact same issue, but only with a specific dataset. Strangely enough, it worked perfectly fine with the other datasets. The inconvenient workaround for me was to recreate the dataset in question.

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	18.04 LTS
Linux Kernel	4.15.0-46-generic
Architecture	x86_64
ZFS Version	0.7.5-1ubuntu16.4
SPL Version	0.7.5-1ubuntu1

[chris13524]

I'm also experiencing this issue. I had followed this guide to setup my system.

I have pool/ROOT/ubuntu mounted on / and pool/home mounted on /home and pool/home/username mounted on /home/username. However, only /.zfs exists not /home/.zfs or /home/username/.zfs even though those are "filesystem roots".

Also running Ubuntu v18.04.3.

[chris13524]

I was unable to reproduce this with a new test pool. Perhaps it has to do with some flag or mount point combination?

[rakor]

I have zfs-on-root installed with this guide: https://github.com/zfsonlinux/zfs/wiki/Debian-Buster-Root-on-ZFS
Its installed using debian-testing and zfs-0.8.2-2.
Inside the boot-pool .zfs/snapshots are working as expected, but in the root-pool its not working.
Inside /.zfs/snapshot are folders with the names of the snapshots, but there folders are empty.
Using 'mount -t zfs ...' I can mount the snapshots, so the data is there, but not using .zfs/snapshot.

[behlendorf]

@rakor it looks like you're hitting #9381 which was recently resolved in by #9384. We'll get this fix applied to 0.8.3, sorry about the inconvenience.

[rakor]

Will I have to recreate the pool?

[behlendorf]

@rakor no, there's wouldn't be any need to recreate the pool.

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

[Ifiht]

Sorry, this still seems to be an issue in Ubuntu 20.04 LTS, using

zfs-0.8.3-1ubuntu12.9
zfs-kmod-0.8.3-1ubuntu12.7

Was this fix ever verified?

[devZer0]

could you please retest with zfs 2.0.4 instead of 0.8.3 ?

[carlosaguilarmelchor]

Same issue here. .zfs has the snapshots of zfs-auto-snapshot but not the ones done manually.

Have not tested to upgrade zfs out of the mainstream packages as it I am using it on a critical server.

[Ifiht]

@carlosaguilarmelchor same boat here, really only choice is to wait for the fix to hit main, since I can't start testing and risk the data.