PAM: support the authentication facility

[valpackett]

Motivation and Context

In the system I'm working on, I'd like to exclusively use the ZFS dataset passphrase for user authentication, in order to avoid extra sources of persistent state (in this case /etc/shadow style files). Seems like this could be useful for others too so I'd like to upstream it.

Description

Implement the pam_sm_authenticate method, using the noop argument of lzc_load_key to do a passphrase check without actually loading the key.

This allows using ZFS as the source of truth for user passwords, without storing any password hashes in /etc or using other PAM modules.

Notes for Other PRs

If #13050 is to land after this, it should either extract the "already mounted" check up to pam_sm_open_session or skip that check in noop mode.

How Has This Been Tested?

Adding these to /etc/pam.d/su

auth            sufficient      pam_zfs_key.so          debug homes=testpool prop_mountpoint

session         optional        pam_zfs_key.so          debug homes=testpool prop_mountpoint

and adding a test user with a home directory pointing to an encrypted dataset (using vipw),

then verifying that su uzer correctly checks the passphrase but doesn't mount anything, while su -l uzer (-l for "Simulate a full login" that does open a session) does also mount and unmount the home dataset.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[behlendorf]

@felixdoerre would you mind taking a look at this PR.