Add support for OnRootMismatch fsGroupChangePolicy

[anothertobi]

Workloads using ReadWriteMany volumes with a lot of files (multiple 100k) have long startup times with some storage providers (e.g. csi-driver-nfs). This happens because without securityContext.fsGroupChangePolicy set, the default for many CSI drivers is to always let the kubelet change the permissions and ownership of all files in a volume (kubernetes.io).

Setting securityContext.fsGroupChangePolicy: OnRootMismatch will fix the file ownership and permissions only if there is a root mismatch which should only happen during initial provisioning or after a change to the fsGroup.

Ideally this is set for all workloads, but only if a feature flag is enabled (opt-in) since this is a breaking change.

[shreddedbacon]

You mention this is a breaking change. If this is applied to something that doesn't have the csi-driver-nfs and is working fine today, and this is applied, what breaks, or how does it break?

[anothertobi]

One breaking case would be that another process mounts the same volume or a subdir with a different permission set and changes file ownership in places that OnRootMismatch doesn't check.

Let's say someone uses csi-driver-nfs with a workload. Now to deploy things they connect to the NFS share with their desktop and upload a few files in a subdir. With the current default, restarting the workload would fix the file permissions the upload introduced. If the option would be set to OnRootMismatch, the kubelet wouldn't fix the ownership and permissions and leave the new files in a state where the workload can't access/modify them.

[shreddedbacon]

What about someone using efs-provisioner, or ceph provisioner, or portworx, or open-nfs. Some other storage provisioner?