node-driver-registrar failed to start in v1.9.0 installation

[marshell0]

What happened:
download csi-driver-smb-1.9.0.tar.gz and execute
csi-driver-smb-1.9.0/deploy/uninstall-driver.sh v1.9.0 local

$ kubectl get po -n kube-system
csi-smb-controller-6497f8ddd7-fp2bg            3/3     Running            0                87m
csi-smb-node-j7g46                                         2/3     CrashLoopBackOff   21 (4m43s ago)   87m
csi-smb-node-m6gjn                                       2/3     CrashLoopBackOff   21 (5m ago)      87m
csi-smb-node-q42wc                                       2/3     CrashLoopBackOff   21 (5m2s ago)    87m
csi-smb-node-v6zxg                                         2/3     CrashLoopBackOff   22 (21s ago)     87m

$ kc logs  -n kube-system  csi-smb-node-v6zxg  | more
Defaulted container "liveness-probe" out of: liveness-probe, node-driver-registrar, smb
I0901 06:12:59.809128       1 connection.go:154] Connecting to unix:///csi/csi.sock
I0901 06:13:00.810583       1 main.go:149] calling CSI driver to discover driver name
I0901 06:13:00.810610       1 connection.go:183] GRPC call: /csi.v1.Identity/GetPluginInfo
I0901 06:13:00.810615       1 connection.go:184] GRPC request: {}
I0901 06:13:00.816749       1 connection.go:186] GRPC response: {"name":"smb.csi.k8s.io","vendor_version":"v1.9.0"}
I0901 06:13:00.816829       1 connection.go:187] GRPC error: <nil>
I0901 06:13:00.816861       1 main.go:155] CSI driver name: "smb.csi.k8s.io"
I0901 06:13:00.816872       1 main.go:183] ServeMux listening at "0.0.0.0:29643"
I0901 06:13:29.331863       1 connection.go:154] Connecting to unix:///csi/csi.sock
I0901 06:13:29.333016       1 main.go:69] Sending probe request to CSI driver "smb.csi.k8s.io"
I0901 06:13:29.333035       1 connection.go:183] GRPC call: /csi.v1.Identity/Probe
I0901 06:13:29.333047       1 connection.go:184] GRPC request: {}
I0901 06:13:29.333931       1 connection.go:186] GRPC response: {"ready":{"value":true}}
I0901 06:13:29.334041       1 connection.go:187] GRPC error: <nil>
I0901 06:13:29.334048       1 main.go:87] Health check succeeded
I0901 06:13:59.331421       1 connection.go:154] Connecting to unix:///csi/csi.sock


$ kc describe pods csi-smb-node-j7g46  -n kube-system
Name:                 csi-smb-node-j7g46
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 bauhinia.mydomain.com/172.17.30.31
Start Time:           Thu, 01 Sep 2022 14:12:59 +0800
Labels:               app=csi-smb-node
                      controller-revision-hash=664645bc4c
                      pod-template-generation=1
Annotations:          <none>
Status:               Running
IP:                   172.17.30.31
IPs:
  IP:           172.17.30.31
Controlled By:  DaemonSet/csi-smb-node
Containers:
  liveness-probe:
    Container ID:  containerd://a05b14f4721f160fd58eebeb220cf988230af1215b3df65b18973a584cf3d0a3
    Image:         nexus.mydomain.com:9443/sig-storage/livenessprobe:v2.7.0
    Image ID:      nexus.mydomain.com:9443/sig-storage/csi-node-driver-registrar@sha256:286a98e5f5df1c8bd8bc2a43702525080bb8a12385f1b2615d178ee62a4f509f
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=/csi/csi.sock
      --probe-timeout=3s
      --health-port=29643
      --v=9
    State:          Running
      Started:      Thu, 01 Sep 2022 14:12:59 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  100Mi
    Requests:
      cpu:        10m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7dxlw (ro)
  node-driver-registrar:
    Container ID:  containerd://c480a9bac429df9c3e2c98486db21b48e715681c5e5891609d13a354508f104e
    Image:         nexus.mydomain.com:9443/sig-storage/csi-node-driver-registrar:v2.5.1
    Image ID:      nexus.mydomain.com:9443/sig-storage/csi-node-driver-registrar@sha256:286a98e5f5df1c8bd8bc2a43702525080bb8a12385f1b2615d178ee62a4f509f
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=$(ADDRESS)
      --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
      --v=9
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Thu, 01 Sep 2022 15:46:06 +0800
      Finished:     Thu, 01 Sep 2022 15:46:06 +0800
    Ready:          False
    Restart Count:  23
    Limits:
      memory:  100Mi
    Requests:
      cpu:     10m
      memory:  20Mi
    Liveness:  exec [/csi-node-driver-registrar --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH) --mode=kubelet-registration-probe] delay=30s timeout=15s period=10s #success=1 #failure=3
    Environment:
      ADDRESS:               /csi/csi.sock
      DRIVER_REG_SOCK_PATH:  /var/lib/kubelet/plugins/smb.csi.k8s.io/csi.sock
    Mounts:
      /csi from socket-dir (rw)
      /registration from registration-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7dxlw (ro)
  smb:
    Container ID:  containerd://e7a2376cf2e597bb09f310776c3eda76a38578ec163d856bbbf824b351e20049
    Image:         nexus.mydomain.com:9443/sig-storage/smbplugin:v1.9.0
    Image ID:      nexus.mydomain.com:9443/sig-storage/smbplugin@sha256:6f73766034258bcc25c4d053b40486eab57a8c9b884b51865b921dd6ed879aef
    Port:          29643/TCP
    Host Port:     29643/TCP
    Args:
      --v=9
      --endpoint=$(CSI_ENDPOINT)
      --nodeid=$(KUBE_NODE_NAME)
      --metrics-address=0.0.0.0:29645
    State:          Running
      Started:      Thu, 01 Sep 2022 14:13:00 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  200Mi
    Requests:
      cpu:     10m
      memory:  20Mi
    Liveness:  http-get http://:healthz/healthz delay=30s timeout=10s period=30s #success=1 #failure=5
    Environment:
      CSI_ENDPOINT:    unix:///csi/csi.sock
      KUBE_NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /csi from socket-dir (rw)
      /var/lib/kubelet/ from mountpoint-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7dxlw (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  socket-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/smb.csi.k8s.io
    HostPathType:  DirectoryOrCreate
  mountpoint-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/
    HostPathType:  DirectoryOrCreate
  registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  DirectoryOrCreate
  kube-api-access-7dxlw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason   Age                    From     Message
  ----     ------   ----                   ----     -------
  Warning  BackOff  3m18s (x430 over 93m)  kubelet  Back-off restarting failed container

What you expected to happen:
csi-smb-node all start normally without failed container

How to reproduce it:
just execute above install command
Anything else we need to know?:
istio installed
Environment:

• CSI Driver version:
  v1.9.0
• Kubernetes version (use kubectl version):
  Client Version: v1.24.0
  Kustomize Version: v4.5.4
  Server Version: v1.24.0
• OS (e.g. from /etc/os-release):
  PRETTY_NAME="openEuler 21.09"
• Kernel (e.g. uname -a):
  Linux bauhinia 5.10.0-5.10.0.24.oe1.x86_64 test: fix travis config #1 SMP Wed Sep 29 19:53:50 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
  kubeadm, 3 control-planes, 1 worker node
• Others:
• istioctl version
  client version: 1.14.1
  control plane version: 1.14.1
  data plane version: 1.14.1 (1 proxies)

[andyzhangx]

is kubelet path correct in your cluster /var/lib/kubelet/? and what's the registrar logs? k logs -n kube-system csi-smb-node-v6zxg -p -c node-driver-registrar

[marshell0]

@andyzhangx Thanks for your prompt response.
The above commands output as below:

$ sudo ls /var/lib/kubelet/
config.yaml  cpu_manager_state	device-plugins	kubeadm-flags.env  memory_manager_state  pki  plugins  plugins_registry  pod-resources	pods
$ sudo ls /var/lib/kubelet/plugins/
smb.csi.k8s.io

$ kc logs -n kube-system csi-smb-node-28cz4 -c  node-driver-registrar
flag provided but not defined: -kubelet-registration-path
Usage of /livenessprobe:
  -add_dir_header
    	If true, adds the file directory to the header of the log messages
  -alsologtostderr
    	log to standard error as well as files
  -csi-address string
    	Address of the CSI driver socket. (default "/run/csi/socket")
  -health-port 9808
    	(deprecated) TCP ports for listening healthz requests. The default is 9808. If set, `--http-endpoint` cannot be set. (default "9808")

[marshell0]

I moved to previous version 2.5.0 from hub.docker.com: objectscale/csi-node-driver-registrar:v2.5.0 (just changed image url of csi-smb-node.yaml v1.9.0), everything moves well right now.

csi-smb-node-5ktxx 3/3 Running 0 3m57s
csi-smb-node-6r75b 3/3 Running 0 3m57s
csi-smb-node-p9v8g 3/3 Running 0 3m57s
csi-smb-node-tj4x9 3/3 Running 0 3m57s

$ kc logs csi-smb-node-6r75b -n kube-system -c node-driver-registrar
I0902 07:35:28.674550 1 main.go:166] Version: v2.5.0
I0902 07:35:28.674650 1 main.go:167] Running node-driver-registrar in mode=registration
I0902 07:35:28.676395 1 main.go:191] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0902 07:35:28.676474 1 connection.go:154] Connecting to unix:///csi/csi.sock
I0902 07:35:29.678854 1 main.go:198] Calling CSI driver to discover driver name
I0902 07:35:29.678946 1 connection.go:183] GRPC call: /csi.v1.Identity/GetPluginInfo
I0902 07:35:29.678960 1 connection.go:184] GRPC request: {}
I0902 07:35:29.690147 1 connection.go:186] GRPC response: {"name":"smb.csi.k8s.io","vendor_version":"v1.9.0"}
I0902 07:35:29.690328 1 connection.go:187] GRPC error:
I0902 07:35:29.690356 1 main.go:208] CSI driver name: "smb.csi.k8s.io"
I0902 07:35:29.690422 1 node_register.go:53] Starting Registration Server at: /registration/smb.csi.k8s.io-reg.sock
I0902 07:35:29.693145 1 node_register.go:62] Registration Server started at: /registration/smb.csi.k8s.io-reg.sock
I0902 07:35:29.694128 1 node_register.go:92] Skipping HTTP server because endpoint is set to: ""
I0902 07:35:30.604933 1 main.go:102] Received GetInfo call: &InfoRequest{}
I0902 07:35:30.605604 1 main.go:109] "Kubelet registration probe created" path="/var/lib/kubelet/plugins/smb.csi.k8s.io/registration"
I0902 07:35:30.687339 1 main.go:120] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}

and above with issue version 2.5.1 I obtained from: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1

And I checked version 2.5.1 of node-driver-registrar golang source code, looks no problem with the kubeletRegistrationPath of flag code portion

[andyzhangx]

does this image work? mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.5.1

[marshell0]

It works, then I double checked my image, and found that I used a wrong link (or tag) of csi-node-driver-registrar.
Because of the firwall policy in China, I have to pull the image from desktop via VPN and tag it, then push to our private repository, looks I tagged a wrong image.
I am so sorry about that and really appreicate your time and your help, thanks a lot.