rolling back partially deleted (no mount points) datasets

[danboid]

Hi Jim

I use syncoid to back up a few Ubuntu (20.04) servers that I maintain. A couple of these are configured with a separate ZFS dataset for each users home directory. Something happened to one of these servers recently that caused all of the directories in /home to get deleted but their datasets were not destroyed and all of them still had several snapshots intact.

My first instinct was to try using findoid to search for a file that I knew existed in one of these home dirs that still had snapshots. I don't have the errrors to hand unfortunately but findoid doesn't seem to work when the dir that the dataset was mounted in no longer exists. This is probably a known limitation of findoid but I didn't know it wouldn't work in this situation.

I could still list the snapshots for the various home dirs so I tried using zfs rollback -r to revert a few of them but this didn't work, zfs didn't seem to recreate the now missing mount point for the reverted snapshot, or at least I couldn't see the files I was expecting to see get restored.

Is this a limitation of ZFS or should this have worked if I had recreated the mount point first before using rollback? Is there any way to automate the recreation of mount points when using zfs rollback or am I using the wrong command(s) to do this?

Also, is there an easy way to rollback multiple (all) datasets in a pool by a given amount of time?

Thanks

[danboid]

I don't really know what my issue was here now because I have just tried recreating a similar config in a Ubuntu server 20.04 VM and when I rolled back two datasets that had had their mount points deleted, zfs rollback -r successfully recreated them and findoid was still working.

My question about rolling back all datasets within a pool (or rolling back all datasets within a parent dataset) by a given amount of time still stands tho. Has anyone written a script to do this that works with snapshots using sanoids snapshot naming scheme?

[danboid]

syncoid is still in the process of restoring all 775 home dirs but I'm not too confident its going to work when the sync is finished because nothing is showing up in /home yet (hopefully that will get fixed by a reboot / re-import of the pool?) and also findoid is still not working on the server getting restored.

If I try to run findoid to search for a file that exists in one of the datasets thats already been transferred back (zfs list shows it as being present) I get the error:

readdir() attempted on invalid dirhandle $dh at /usr/sbin/findoid line 93.
closedir() attempted on invalid dirhandle $dh at /usr/sbin/findoid line 96.

zpool status says there are no errors on any disks in its pool so do you have any suggestions what might be wrong?

Is there an known limit as to how many datasets sanoid can create and manage snapshots for before ZFS taps out?

The sanoid config for this server with 775 home dir datasets looks like:

[astarray/home]
	use_template = production
        recursive = yes

#############################
# templates below this line #
#############################

[template_production]
        frequently = 4
        hourly = 24
        daily = 30
        weekly = 12
        monthly = 3
        yearly = 0
        autosnap = yes
        autoprune = yes

It is a RAIDZ2 pool consisting of 6x SAMSUNG 870 SATA 3 SSDs. I destroyed the parent dataset for /home before I ran syncoid to give it the best chance of restoring successfully but it looks like that might've not been enough.

[lckarssen]

nothing is showing up in /home yet

I'm not sure newly created/restored ZFS datasets will automatically be mounted. Instead of looking in /home, try to run zfs list -r POOLNAME to see if the datasets of the home directories appear there.

[danboid]

I didn't think syncoid was working because I didn't know until today that it doesn't restore dataset mount points by default so it was mounting them in the wrong place.