Let zfs mount all tolerate in-progress mounts

[don-brady]

Motivation and Context

The zfs-mount service can unexpectedly fail to start when zfs encounters a mount that is in progress. This service uses zfs mount -a, which has a window between the time it checks if the dataset was mounted and when the actual mount (via mount.zfs binary) occurs.

Description

Simple way to demonstrate this in-progress mount window:

$ for i in {1..20}; do sudo zfs create dozer/ds$i; done
$ sudo zfs unmount dozer/ds20
$ sudo zfs mount -a & sudo zfs mount -a &
[1] 8568
[2] 8569
$
filesystem 'dozer/ds20' is already mounted
cannot mount 'dozer/ds20': mountpoint or dataset is busy
[1]-  Exit 1                  sudo cmd/zfs/zfs mount -a
[2]+  Done                    sudo cmd/zfs/zfs mount -a

The suggested solution is to check if the failure was due to an in-progress mount and not treat it as an error.

How Has This Been Tested?

Manually tested the simple case described above to confirm zfs mount -a is working as intended.

Tested with a suite of automated test that was routinely encountering this error. A journalctl audit confirms that the zfs-mount service no longer fails to start when it encounters an in-progress mount.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the ZFS on Linux code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• All commit messages are properly formatted and contain Signed-off-by.

[codecov]

Codecov Report

Merging #8881 into master will increase coverage by 0.1%.
The diff coverage is 50%.

@@            Coverage Diff            @@
##           master    #8881     +/-   ##
=========================================
+ Coverage   78.65%   78.76%   +0.1%     
=========================================
  Files         382      382             
  Lines      117791   117785      -6     
=========================================
+ Hits        92652    92771    +119     
+ Misses      25139    25014    -125

Flag	Coverage Δ
#kernel	79.32% <ø> (+0.06%)	⬆️
#user	67.49% <50%> (+0.44%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b873825...461db88. Read the comment docs.

[behlendorf]

I think including this additional resiliency makes good sense. But I'd like to better understand why you were seeing concurrent mounts at all. zfs mount -a should only be run once by the zfs-mount service. Which is conflicting with an unrelated zfs mount in your environment?

[behlendorf]

As I understand it, the reason we're seeing these racing mounts is that both the zfs-mount.target and zfs-share.target are allowed to execute concurrently after the import. This is likely more of an issue with the relatively recent addition of parallel mounting.

@aerusso I was hoping to get your thoughts of the best way to handle this from a systemd perspective. We used to run the zfs-share.target as After=zfs-mount.service, but it was converted to Wants=zfs-mount.service in commit 80b4852. Does it make sense to consider converting it back, or will this break some other use case?