microk8s + kerberos integration issues

[fabioscaccabarozzi]

What happened: when trying to setup PV+PVC with Kerberos under microk8s, nodes cannot mount the share, and pods stay in "ContainerCreating" state, until they fail and are restarted.

What you expected to happen: share is mounted fine, pods can correctly start

How to reproduce it:

Steps to reproduce
Assuming we're on a host where Kerberos is configured, and cifs-utils and keyutils are installed.
Install microk8s via snap: snap install microk8s --classic
Install CSI driver with values supplied below.
Use the driver-parameters docs for the kerberos part (kinit, secret, ...)
Create the resources below. PV and PVC will show "Bound".
Attempt to create a deployment that uses the PVC (see example below).
Whichever node attempts to run the pod will fail with the following error:

microk8s.daemon-kubelite[1422]: E1212 11:06:34.115973    1422 csi_attacher.go:364] kubernetes.io/csi: attacher.MountDevice failed: rpc error: code = Internal desc = Error writing kerberos cache: rpc error: code = Internal desc = Directory for kerberos caches must exist, it will not be created: /var/lib/kubelet/kerberos/: stat /var/lib/kubelet/kerberos/: no such file or directory

---

Investigation & Explanation
In microk8s, by default /var/lib/kubelet is a symlink pointing to /var/snap/microk8s/common/var/lib/kubelet.
Kerberos was configured according to the driver-parameter docs linked above, so we have:

[libdefaults]
        debug = true
        default_realm = <realm.com>
        default_ccache_name = FILE:/var/lib/kubelet/kerberos/krb5cc_%{uid}
        kdc_timesync = 1
        ccache_type = 4
        forwardable = true
        proxiable = true

[realms]
...
[domain_realm]
...

The first problem is that if we supply linux.kubelet at chart installation, that folder will be mounted by the chart in the csi-smb-node pod verbatim via hostPath (host: /path/A -> pod: /path/A).

The second problem is we can't seem to find a way to alter the Kerberos cache path. This means that if we move the kubelet path via linux.kubelet, the /var/lib/kubelet path won't exist in the csi-smb-node pods, making it impossible to create and save kerberos tickets from the pod to the host system, which ultimately is the one performing the mounts via cifs-utils.

Further, currently in the Kerberos cache folder absolute symlinks are used from krb5cc_UID files to the actual token files (e.g: /var/lib/kubelet/kerberos/krb5cc_1000 -> /var/lib/kubelet/kerberos/aW1nLXNoYXJlLWRldm1xcw==), so the source path from the host must be exactly the same also in the pod, otherwise the absolute symlinks won't work for cifs-utils.

As it stands now, csi-smb-node expects the the kerberos cache to be at /var/lib/kubelet/kerberos and nowhere else, making it impossible the use of Kerberos with a different linux.kubelet.

---

Fixing attempt
I have tried to modify the helm chart template for csi-smb-node to mount the Kerberos cache folder as an additional volume, or the same volume with subPath, but it doesn't work -> touch /var/lib/..../kerberos/test would yield no such file or directory, I guess because I cannot mount the same path twice for the same pod. Anyway that would clash with the hard-coded path in the file referenced above.

---

Proposed solution
The better solution would be to make the Kerberos cache path configurable as the linux.kubelet path is, and ideally discourage the use of /var/lib/kubelet/kerberos, as in microk8s that path is a symlink to the snap folder.

---

Objects used
The helm chart values, as per docs for microk8s:

$ helm get values -nkube-system csi-driver-smb
USER-SUPPLIED VALUES:
linux:
  kubelet: /var/snap/microk8s/common/var/lib/kubelet

These are the credentials, PV and PVC I'm trying to use:

apiVersion: v1
kind: Secret
stringData:
  krb5cc_1000: __SMB_SHARE_KRB5CC__
metadata:
  name: smbcreds-krb5
  namespace: proj-ns
type: Opaque
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-smb-krb5
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: smb
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - noperm
    - mfsymlinks
    - cache=strict
    - noserverino # required to prevent data corruption
    - sec=krb5
    - uid=1000
    - cruid=1000
  csi:
    driver: smb.csi.k8s.io
    readOnly: false
    volumeHandle: pv-proj-ns-krb5 # make sure it's a unique id in the cluster
    volumeAttributes:
      source: "__SMB_SHARE_URL__"
    nodeStageSecretRef:
      name: smbcreds-krb5
      namespace: proj-ns
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-smb-krb5
  namespace: proj-ns
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  volumeName: pv-smb-krb5
  storageClassName: smb

An example deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: proj-deploy
  namespace: proj-ns
spec:
  selector:
    matchLabels:
      name: proj-deploy
  replicas: 2
  revisionHistoryLimit: 0
  template:
    metadata:
      labels:
        name: proj-deploy
    spec:
      restartPolicy: Always
      volumes:
        - name: smb
          persistentVolumeClaim:
            claimName: pvc-smb-krb5
      containers:
        - name: test-deploy
          image: nginx
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - mountPath: /data
              name: smb

Anything else we need to know?:

Environment:

• CSI Driver version: csi-driver-smb-v1.13.0
• Kubernetes version (use kubectl version): 1.27.8 (snap)
• OS (e.g. from /etc/os-release): Ubuntu 20.04 LTS
• Kernel (e.g. uname -a): 5.4.0-169-generic

[andyzhangx]

@fabioscaccabarozzi current kerberos path is /var/lib/kubelet/kerberos, what's the expected path in your env? is it /var/snap/microk8s/common/var/lib/kubelet/kerberos?

anyway, let me make it configurable first.

[fabioscaccabarozzi]

@andyzhangx I do not have an expected path - I followed the guides and bits found for microk8s and for kerberos, and I finally realized that the two cannot work together at the moment.

I guess that either providing a configurable path indipendent of linux.kubelet (e.g: host: /var/lib/kerberos, pod: /var/lib/kerberos) or keeping the kerberos path relative to the linux.kubelet path, e.g:

• linux.kubelet: /var/snap/microk8s/common/var/lib/kubelet
• host: /var/snap/microk8s/common/var/lib/kubelet/kerberos
• pod: /var/snap/microk8s/common/var/lib/kubelet/kerberos)

should work.

[andyzhangx]

so if I set /var/lib/kubelet/kerberos as configurable in this csi driver, it shoud work @fabioscaccabarozzi ?

[fabioscaccabarozzi]

yes, that's my expectation -> if the CSI driver and the host kerberos/cifs-utils can use the same cache folder with the proper symlinks, everything should work

[andyzhangx]

#709 could fix this issue

[fabioscaccabarozzi]

The PR looks good to me, I believe it's the right approach.

One minor nitpick: as it is now, if linux.krb5CacheDirectory is not a sub-path of linux.kubelet, the csi-smb-node pod won't have the mount to share with the host.

I believe we need to add an additional volume here in case users specify something like:

• linux.kubelet: /var/snap/microk8s/common/var/lib/kubelet
• linux.krb5CacheDirectory: /var/lib/kerberos

How can I test the PR? Is there some image.*.tag override I can use in the helm chart values?

I should be able to start testing it tomorrow.

Thank you!

[andyzhangx]

@fabioscaccabarozzi can you move kerberos to /var/snap/microk8s/common/var/lib/kubelet/kerberos path, and then set --krb5-cache-directory=/var/snap/microk8s/common/var/lib/kubelet/kerberos with image andyzhangx/smb-csi:v1.14.0

https://github.com/kubernetes-csi/csi-driver-smb/blob/4a84e512499af588303fb2e31b9b6df547f50538/charts/latest/csi-driver-smb/templates/csi-smb-node.yaml#L111C16-L111C39

[andyzhangx]

wait, I don't think this PR would work since /var/lib/kubelet/kerberos/ is hardcoded in /etc/krb5.conf in the driver image, I think this requires another PR to map kublet path to /var/lib/kubelet/ inside driver.

RUN echo "[libdefaults]\n    default_ccache_name = FILE:/var/lib/kubelet/kerberos/krb5cc_%{uid}\n" > /etc/krb5.conf

https://github.com/kubernetes-csi/csi-driver-smb/blob/4a84e512499af588303fb2e31b9b6df547f50538/cmd/smbplugin/Dockerfile#L25C116-L25C116

[fabioscaccabarozzi]

Ah, you're right, I didn't even see that part, sorry.

Yes, agree that part also needs to be changed.

Perhaps it could be mounted via a configmap via the helm chart?

[andyzhangx]

#711 should fix this issue, so you don't need to change any setting, as long as kerberos path is /var/snap/microk8s/common/var/lib/kubelet/kerberos, it should work since inside driver, it's mapping to /var/lib/kubelet/kerberos

[fabioscaccabarozzi]

#711 will not fix everything -> kerberos utils will create cache files in /var/lib/kubelet/kerberos with absolute symlinks pointing to /var/lib/kubelet/kerberos/<file>, but if host path is mounted from, e.g: /var/lib/kerberos, the host cifs-utils won't be able to use them.

[andyzhangx]

#714 could fix this issue, this PR adds a mount path for krb5CacheDirectory if it's not empty

[fabioscaccabarozzi]

This fixes the mount+symlinks, the only outstanding part now is the /etc/krb5.conf

This can be done by via the helm chart, using a ConfigMap to mount the file with the custom config. This way the container image contains the default, and gets overridden if required.

[andyzhangx]

This fixes the mount+symlinks, the only outstanding part now is the /etc/krb5.conf

This can be done by via the helm chart, using a ConfigMap to mount the file with the custom config. This way the container image contains the default, and gets overridden if required.

@fabioscaccabarozzi what about this fix: #715 ?
inside the driver, the path is always /var/lib/kubelet/kerberos, that would work?