node-driver-registrar in CrashLoopBackOff status following upgrade to v1.14.0

[sherif-fanous]

What happened:

Upgraded Helm Chart to v1.14.0

node-driver-registrar is in CrashLoopBackOff with the following logs

I0117 18:51:30.843865       1 main.go:135] Version: v2.9.1
I0117 18:51:30.843925       1 main.go:136] Running node-driver-registrar in mode=
I0117 18:51:30.843929       1 main.go:157] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0117 18:51:30.845275       1 main.go:164] Calling CSI driver to discover driver name
I0117 18:51:30.846750       1 main.go:173] CSI driver name: "smb.csi.k8s.io"
I0117 18:51:30.846872       1 node_register.go:55] Starting Registration Server at: /registration/smb.csi.k8s.io-reg.sock
I0117 18:51:30.846946       1 node_register.go:64] Registration Server started at: /registration/smb.csi.k8s.io-reg.sock
I0117 18:51:30.847029       1 node_register.go:88] Skipping HTTP server because endpoint is set to: ""
I0117 18:51:31.929033       1 main.go:90] Received GetInfo call: &InfoRequest{}
I0117 18:51:31.932529       1 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins/smb.csi.k8s.io/csi.sock: connect: no such file or directory",}
E0117 18:51:31.932548       1 main.go:103] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins/smb.csi.k8s.io/csi.sock: connect: no such file or directory", restarting registration container.

What you expected to happen:

Upgrade to succeed and all pods to be in Running status

Anything else we need to know?:

This issue is as a result of #711 as this now hardcodes the kubelet path in the helm chart but quite a few kubernetes distributions don't use /var/lib/kubelet as their kubelet path

In my case I'm using k0s and the kubelet location is /var/lib/k0s/kubelet

Environment:

• CSI Driver version: v1.14.0
• Kubernetes version (use kubectl version): Server Version: v1.28.4+k0s
• OS (e.g. from /etc/os-release): Debian GNU/Linux 12 (bookworm)
• Kernel (e.g. uname -a): Linux k0s 6.1.0-16-arm64 test: fix travis config #1 SMP Debian 6.1.67-1 (2023-12-12) aarch64 GNU/Linux

[sherif-fanous]

A workaround to fix the issue is to create a sym link

ln -s /var/lib/k0s/kubelet/ /var/lib/kubelet

[andyzhangx]

@sherif-fanous have you set --set linux.kubelet="/var/lib/k0s/kubelet/" in helm install, if so, /var/lib/k0s/kubelet/ would be mapped to /var/lib/kubelet inside the csi driver.

[sherif-fanous]

I have using a values file but without the sym link I created I was getting the error I shared in the first comment @andyzhangx

linux:
  kubelet: /var/lib/k0s/kubelet

[brunnels]

This was broken here:
https://github.com/kubernetes-csi/csi-driver-smb/blob/master/charts/v1.14.0/csi-driver-smb/templates/csi-smb-node.yaml#L90

It was right in v1.13.0 https://github.com/kubernetes-csi/csi-driver-smb/blob/master/charts/v1.13.0/csi-driver-smb/templates/csi-smb-node.yaml#L90

[andyzhangx]

fixed by #741 on v1.14.0 helm chart