Hetzner volumes won't get created

[Lapotor]

I created a new Cluster a few days back.
Now I try to add Pods with PVC as hvolumes.

Put PVC keep pending and no volume got created so far.

I traced the creations of the csi-hetzner and default-storage-class files everything seems that i got created, but the Volumes are pending.

[kron4eg]

@Lapotor please take a look at kubectl describe for your PVC.

[Lapotor]

@kron4eg sure here is the output:

$ kubectl describe -n erpnext pvc csi-pvc
Name:          csi-pvc
Namespace:     erpnext
StorageClass:  hcloud-volumes
Status:        Pending
Volume:
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age                From                         Message
  ----    ------                ----               ----                         -------
  Normal  WaitForFirstConsumer  10s (x3 over 33s)  persistentvolume-controller  waiting for first consumer to be created before binding

[xmudrii]

@Lapotor Can you please share:

• Your KubeOneCluster manifest (kubeone.yaml)
• Output of kubectl get nodes -o wide
• Output of kubectl get pods -n kube-system
• Output of kubectl get sc

Please make sure to redact any secrets if there are any.

[Lapotor]

Sure @xmudrii

kubeone.yaml

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
  kubernetes: '1.24.3'
cloudProvider:
  hetzner: {}
  external: true
addons:
  enable: true
  addons:
  - name: cluster-autoscaler

$ kubectl get nodes -o wide
NAME                            STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP     OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
<name>-control-plane-1         Ready    control-plane   16d   v1.24.3   <Internal-IP> <External-IP>    Ubuntu 20.04.5 LTS   5.4.0-125-generic   containerd://1.6.9
<name>-control-plane-2         Ready    control-plane   16d   v1.24.3   <Internal-IP> <External-IP>    Ubuntu 20.04.5 LTS   5.4.0-125-generic   containerd://1.6.9
<name>-pool1-79ddf6f4c-4cwdm   Ready    <none>          16d   v1.24.3   <Internal-IP> <External-IP>    Ubuntu 20.04.5 LTS   5.4.0-125-generic   containerd://1.6.9

$ kubectl get pods -n kube-system
NAME                                                READY   STATUS    RESTARTS   AGE
calico-kube-controllers-77559f8ddc-789jv            1/1     Running   0          16d
canal-25rrz                                         2/2     Running   0          16d
canal-nv444                                         2/2     Running   0          16d
canal-zf4lg                                         2/2     Running   0          16d
cluster-autoscaler-c9b79c6dd-xjxsp                  1/1     Running   0          16d
coredns-56fc8c7fff-b9x9b                            1/1     Running   0          16d
coredns-56fc8c7fff-smv7j                            1/1     Running   0          16d
etcd-jbm-k8s-control-plane-1                        1/1     Running   0          16d
etcd-jbm-k8s-control-plane-2                        1/1     Running   0          16d
hcloud-cloud-controller-manager-65c55df974-fhxgb    1/1     Running   0          16d
hcloud-csi-controller-0                             5/5     Running   0          16d
hcloud-csi-node-8dhvg                               3/3     Running   0          16d
hcloud-csi-node-8wt4m                               3/3     Running   0          16d
hcloud-csi-node-w6kh4                               3/3     Running   0          16d
kube-apiserver-jbm-k8s-control-plane-1              1/1     Running   1          16d
kube-apiserver-jbm-k8s-control-plane-2              1/1     Running   0          16d
kube-controller-manager-jbm-k8s-control-plane-1     1/1     Running   0          16d
kube-controller-manager-jbm-k8s-control-plane-2     1/1     Running   0          16d
kube-proxy-89rhn                                    1/1     Running   0          16d
kube-proxy-8f9xl                                    1/1     Running   0          16d
kube-proxy-gbql9                                    1/1     Running   0          16d
kube-scheduler-jbm-k8s-control-plane-1              1/1     Running   0          16d
kube-scheduler-jbm-k8s-control-plane-2              1/1     Running   0          16d
machine-controller-7f45dcdcdf-jzpfj                 1/1     Running   0          16d
machine-controller-webhook-59d8bfbdcc-hd5tz         1/1     Running   0          16d
metrics-server-6c8f44fb59-72cj5                     1/1     Running   0          16d
node-local-dns-bstrd                                1/1     Running   0          16d
node-local-dns-khdjs                                1/1     Running   0          16d
node-local-dns-vzbr4                                1/1     Running   0          16d
operating-system-manager-d568ddc9-5bwdn             1/1     Running   0          16d
operating-system-manager-webhook-85f547c74b-6rckj   1/1     Running   0          16d

$ kubectl get sc
NAME                       PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
csi-s3                     ch.ctrox.csi.s3-driver   Delete          Immediate              false                  16d
hcloud-volumes (default)   csi.hetzner.cloud        Delete          WaitForFirstConsumer   true                   13h

[xmudrii]

Do you have any pod that is using that PVC? The PersisentVolume will get created only if some pod is using that PVC because the hcloud-volumes StorageClass has VolumeBindingMode set to WaitForFirstConsumer. If you don't have any pods using that PVC, please try to create one.

[Lapotor]

Ok i have no clue, i tired the example from hetznercloud/csi-driver yesterday several times.
Today it works.
Thanks anyway for your help.

[kron4eg]

$ kubectl get nodes -o wide
NAME                            STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP     OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
<name>-control-plane-1         Ready    control-plane   16d   v1.24.3   <Internal-IP> <External-IP>    Ubuntu 20.04.5 LTS   5.4.0-125-generic   containerd://1.6.9
<name>-control-plane-2         Ready    control-plane   16d   v1.24.3   <Internal-IP> <External-IP>    Ubuntu 20.04.5 LTS   5.4.0-125-generic   containerd://1.6.9
<name>-pool1-79ddf6f4c-4cwdm   Ready    <none>          16d   v1.24.3   <Internal-IP> <External-IP>    Ubuntu 20.04.5 LTS   5.4.0-125-generic   containerd://1.6.9

BTW, you have only 2 control plane nodes, it means if one of them fails — your cluster is dead.