Workload problem after upgrade to KubeOne 1.8 and Kubernetes 1.29.4

[clickersmudge]

Hi,
I upgraded KubeOne from version 1.7 to version 1.8 on Hetzner cloud, Ubuntu 22.04.

terraform.tfvars

cluster_autoscaler_max_replicas    = 0
cluster_autoscaler_min_replicas    = 0
cluster_name                       = "prod"
control_plane_type                 = "cpx21"
control_plane_vm_count             = 3
datacenter                         = "nbg1"
# image                              = "103908130"
initial_machinedeployment_replicas = 0
ip_range                           = "10.0.0.0/16"
lb_type                            = "lb11"
network_zone                       = "eu-central"
os                                 = "ubuntu"
ssh_public_key_file                = "~/.ssh/prod.pub"
worker_type                        = "cpx11"

kubeone.yaml

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
  kubernetes: "v1.29.4"
clusterNetwork:
  kubeProxy:
    ipvs:
      scheduler: rr
      strictARP: false
      tcpTimeout: "0"
      tcpFinTimeout: "0"
      udpTimeout: "0"
      excludeCIDRs: []
  cni:
    canal:
      mtu: 1400
cloudProvider:
  hetzner: {}
  external: true
features:
  encryptionProviders:
    enable: false
addons:
  enable: true
  path: "./addons"
  addons:
    - name: "backups-restic"

Pods on the control plane work.
The problem is workload.
The machinedeployment is not working.
Node is created and added correctly. It adds to the firewall and network.

kubectl describe pod -n actions-runner-system           actions-runner-controller-5799fd678f-7mr9c 
  Normal   Scheduled               33m                    default-scheduler   Successfully assigned actions-runner-system/actions-runner-controller-5799fd678f-7mr9c to prod-worker-arm-5c4c8fc554-rptwp
  Warning  FailedCreatePodSandBox  33m                    kubelet             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8befc1671407f3ae4dd5259eda7eed04a4b3a6c740e36c480c5efae70c81da18": plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
  Normal   SandboxChanged          3m42s (x140 over 33m)  kubelet             Pod sandbox changed, it will be killed and re-created.

kubectl describe pod -n argocd                          argocd-argo-cd-app-controller-77fffd8759-xfv5n

  Warning  FailedScheduling        50m (x3 over 60m)      default-scheduler   0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   NotTriggerScaleUp       43m (x122 over 63m)    cluster-autoscaler  pod didn't trigger scale-up:
  Warning  FailedScheduling        34m (x5 over 70m)      default-scheduler   0/3 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
  Normal   Scheduled               34m                    default-scheduler   Successfully assigned argocd/argocd-argo-cd-app-controller-77fffd8759-xfv5n to prod-worker-arm-5c4c8fc554-rptwp
  Warning  FailedCreatePodSandBox  34m                    kubelet             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9c2dfd4b2e47348012d151fb3d3eb612d8f1a0475dc279f74e44200aedce0cb7": plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
  Normal   SandboxChanged          4m14s (x142 over 34m)  kubelet             Pod sandbox changed, it will be killed and re-created.

 kubectl describe pod -n kube-system canal-vltw7
  Warning  BackOff           35m (x3 over 35m)    kubelet            Back-off restarting failed container mount-bpffs in pod canal-vltw7_kube-system(1ec47ad1-5cb8-41d7-ba27-439971dc38c8)
  Warning  DNSConfigForming  65s (x177 over 36m)  kubelet            Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 2a01:4ff:ff00::add:1

We can't figure it out. Can we ask for help.

[xmudrii]

It looks like one of your canal pods is stuck. Can you share describe output for that pod?

[clickersmudge]

Thank @xmudrii for a quick response.

kubectl describe pod -n kube-system canal-vltw7
Name:                 canal-vltw7
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      canal
Node:                 prod-worker-arm-5c4c8fc554-rptwp/10.0.0.6
Start Time:           Wed, 29 May 2024 10:30:40 +0200
Labels:               controller-revision-hash=7bb8fffb44
                      k8s-app=canal
                      pod-template-generation=2
Annotations:          <none>
Status:               Pending
IP:                   10.0.0.6
IPs:
  IP:           10.0.0.6
Controlled By:  DaemonSet/canal
Init Containers:
  install-cni:
    Container ID:  containerd://0fdc542551483f43788ffdb70da20fffcc5bb2a91fafcc48afe109eefeea2d3c
    Image:         quay.io/calico/cni:v3.27.2
    Image ID:      quay.io/calico/cni@sha256:d17e907f27956d82e978f4aee8c21f876230955f3fba51da621c7c49e71f2ece
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/install
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 29 May 2024 10:30:46 +0200
      Finished:     Wed, 29 May 2024 10:30:46 +0200
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      CALICO_CNI_SERVICE_ACCOUNT:   (v1:spec.serviceAccountName)
      CNI_CONF_NAME:               10-canal.conflist
      CNI_NETWORK_CONFIG:          <set to the key 'cni_network_config' of config map 'canal-config'>  Optional: false
      KUBERNETES_NODE_NAME:         (v1:spec.nodeName)
      CNI_MTU:                     <set to the key 'veth_mtu' of config map 'canal-config'>  Optional: false
      SLEEP:                       false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-68qkj (ro)
  mount-bpffs:
    Container ID:  containerd://613ef693913d44745d3a8b5951880b06568117708a3f4aed89257915cd6cea10
    Image:         quay.io/calico/node:v3.27.2
    Image ID:      quay.io/calico/node@sha256:0d00b1f4235652b370f3897803d7c8ef5a7059a081ab8d69ae02b7ca0cb0193b
    Port:          <none>
    Host Port:     <none>
    Command:
      calico-node
      -init
      -best-effort
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    127
      Started:      Wed, 29 May 2024 11:32:39 +0200
      Finished:     Wed, 29 May 2024 11:32:39 +0200
    Ready:          False
    Restart Count:  17
    Environment:    <none>
    Mounts:
      /nodeproc from nodeproc (ro)
      /sys/fs from sys-fs (rw)
      /var/run/calico from var-run-calico (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-68qkj (ro)
Containers:
  calico-node:
    Container ID:   
    Image:          quay.io/calico/node:v3.27.2
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      250m
    Liveness:   exec [/bin/calico-node -felix-live] delay=10s timeout=10s period=10s #success=1 #failure=6
    Readiness:  http-get http://localhost:9099/readiness delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      FELIX_IPTABLESBACKEND:              Auto
      CALICO_IPV4POOL_CIDR:               <set to the key 'CALICO_IPV4POOL_CIDR' of config map 'canal-config'>  Optional: true
      FELIX_IPV6SUPPORT:                  <set to the key 'FELIX_IPV6SUPPORT' of config map 'canal-config'>     Optional: true
      IP6:                                <set to the key 'IP6' of config map 'canal-config'>                   Optional: true
      CALICO_IPV6POOL_CIDR:               <set to the key 'CALICO_IPV6POOL_CIDR' of config map 'canal-config'>  Optional: true
      DATASTORE_TYPE:                     kubernetes
      USE_POD_CIDR:                       true
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_CNI_SERVICE_ACCOUNT:          (v1:spec.serviceAccountName)
      CALICO_NETWORKING_BACKEND:          none
      CLUSTER_TYPE:                       k8s,canal
      FELIX_IPTABLESREFRESHINTERVAL:      60
      IP:                                 
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_HEALTHENABLED:                true
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /sys/fs/bpf from bpffs (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/log/calico/cni from cni-log-dir (ro)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-68qkj (ro)
  kube-flannel:
    Container ID:  
    Image:         docker.io/flannel/flannel:v0.21.3
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/bin/flanneld
      --ip-masq
      --kube-subnet-mgr
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      FLANNELD_IFACE:         (v1:status.hostIP)
      FLANNELD_IFACE_REGEX:  <set to the key 'canal_iface_regex' of config map 'canal-config'>  Optional: false
      POD_NAME:              canal-vltw7 (v1:metadata.name)
      POD_NAMESPACE:         kube-system (v1:metadata.namespace)
      FLANNELD_IP_MASQ:      <set to the key 'masquerade' of config map 'canal-config'>  Optional: false
    Mounts:
      /etc/kube-flannel/ from flannel-cfg (rw)
      /run/xtables.lock from xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-68qkj (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 False 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:  
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:  
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  sys-fs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/
    HostPathType:  DirectoryOrCreate
  bpffs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/bpf
    HostPathType:  Directory
  nodeproc:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:  
  flannel-cfg:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      canal-config
    Optional:  false
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:  
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:  
  cni-log-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/calico/cni
    HostPathType:  
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  kube-api-access-68qkj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 :NoSchedule op=Exists
                             :NoExecute op=Exists
                             CriticalAddonsOnly op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason            Age                    From     Message
  ----     ------            ----                   ----     -------
  Warning  DNSConfigForming  3m43s (x298 over 63m)  kubelet  Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 2a01:4ff:ff00::add:1 2a01:4ff:f

[xmudrii]

Can you please share logs for the mount-bpffs container from that pod?

[clickersmudge]

kubectl logs -n kube-system canal-dl96z
Defaulted container "calico-node" out of: calico-node, kube-flannel, install-cni (init), mount-bpffs (init)
Error from server (BadRequest): container "calico-node" in pod "canal-dl96z" is waiting to start: PodInitializing

[xmudrii]

You're affected by this issue #3198. We'll include the fix for this issue in the next 1.8.1 patch release, however, I don't have any ETA when that patch release will be available at this moment. As a short term mitigation, you can manually change the Calico version from v3.27.2 to v3.27.3. For that, you need to change the canal-node DaemonSet and the calico-kube-controllers Deployment, and update relevant images (no other changes are required). However, please note that if you run kubeone apply afterwards, it's going to revert back to v3.27.2, so you'll not be able to run KubeOne until 1.8.1 is not out (or you'll have to apply this mitigation again).

[clickersmudge]

@xmudrii It works. Thanks.