Empty directory /opt/cni
#5610
-
|
Red Hat Enterprise Linux 8.9 has empty directory /opt/cni and pod fails to start. [root@red ~]# /var/lib/rancher/rke2/bin/kubectl describe pods
Name: smartview-0-1710346441-bc78d4575-mskfp
Namespace: default
Priority: 0
Service Account: smartview-0-1710346441
Node: red.portalify.com/10.80.9.125
Start Time: Wed, 13 Mar 2024 18:14:02 +0200
Labels: app.kubernetes.io/instance=smartview-0-1710346441
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=smartview
app.kubernetes.io/version=1.16.0
helm.sh/chart=smartview-0.1.0
pod-template-hash=bc78d4575
Annotations: <none>
Status: Terminating (lasts 103m)
Termination Grace Period: 30s
IP:
IPs: <none>
Controlled By: ReplicaSet/smartview-0-1710346441-bc78d4575
Containers:
smartview:
Container ID:
Image: nginx:1.16.0
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Liveness: http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-49vsj (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-49vsj:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 148m default-scheduler Successfully assigned default/smartview-0-1710346441-bc78d4575-mskfp to red.portalify.com
Warning FailedCreatePodSandBox 148m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "21a35166960a0fd041f3c4b85c323c2880410b544b11ac42b7be8dcdd5804204": plugin type="bridge" failed (add): failed to find plugin "bridge" in path [/opt/cni/bin]
Normal SandboxChanged 113m (x164 over 148m) kubelet Pod sandbox changed, it will be killed and re-created.
Normal SandboxChanged 106m (x26 over 112m) kubelet Pod sandbox changed, it will be killed and re-created.
Warning FailedKillPod 91m (x58 over 104m) kubelet error killing pod: failed to "KillPodSandbox" for "2c92b883-bbdc-4bcb-8e9d-5d739c2ed2b2" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"21a35166960a0fd041f3c4b85c323c2880410b544b11ac42b7be8dcdd5804204\": plugin type=\"loopback\" failed (delete): failed to find plugin \"loopback\" in path [/opt/cni/bin]"
Warning FailedMount 73m (x3 over 73m) kubelet MountVolume.SetUp failed for volume "kube-api-access-49vsj" : failed to fetch token: serviceaccounts "smartview-0-1710346441" not found
Warning FailedMount 73m kubelet MountVolume.SetUp failed for volume "kube-api-access-49vsj" : failed to fetch token: Post "https://127.0.0.1:6443/api/v1/namespaces/default/serviceaccounts/smartview-0-1710346441/token": dial tcp 127.0.0.1:6443: connect: connection refused
Warning FailedMount 73m kubelet MountVolume.SetUp failed for volume "kube-api-access-49vsj" : failed to fetch token: serviceaccounts "smartview-0-1710346441" is forbidden: User "system:node:red.portalify.com" cannot create resource "serviceaccounts/token" in API group "" in the namespace "default": no relationship found between node 'red.portalify.com' and this object
Warning FailedKillPod 73m (x2 over 73m) kubelet error killing pod: failed to "KillPodSandbox" for "2c92b883-bbdc-4bcb-8e9d-5d739c2ed2b2" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"21a35166960a0fd041f3c4b85c323c2880410b544b11ac42b7be8dcdd5804204\": plugin type=\"loopback\" failed (delete): failed to find plugin \"loopback\" in path [/opt/cni/bin]"
Warning FailedMount 72m kubelet MountVolume.SetUp failed for volume "kube-api-access-49vsj" : failed to fetch token: serviceaccounts "smartview-0-1710346441" is forbidden: User "system:node:red.portalify.com" cannot create resource "serviceaccounts/token" in API group "" in the namespace "default": no relationship found between node 'red.portalify.com' and this object
Warning FailedMount 70m (x8 over 72m) kubelet MountVolume.SetUp failed for volume "kube-api-access-49vsj" : failed to fetch token: serviceaccounts "smartview-0-1710346441" not found
Warning FailedKillPod 57m (x71 over 72m) kubelet error killing pod: failed to "KillPodSandbox" for "2c92b883-bbdc-4bcb-8e9d-5d739c2ed2b2" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"21a35166960a0fd041f3c4b85c323c2880410b544b11ac42b7be8dcdd5804204\": plugin type=\"loopback\" failed (delete): failed to find plugin \"loopback\" in path [/opt/cni/bin]"
Warning FailedMount 50m (x9 over 52m) kubelet MountVolume.SetUp failed for volume "kube-api-access-49vsj" : [failed to fetch token: serviceaccounts "smartview-0-1710346441" not found, object "default"/"kube-root-ca.crt" not registered]
Warning FailedKillPod 2m6s (x231 over 52m) kubelet error killing pod: failed to "KillPodSandbox" for "2c92b883-bbdc-4bcb-8e9d-5d739c2ed2b2" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"21a35166960a0fd041f3c4b85c323c2880410b544b11ac42b7be8dcdd5804204\": plugin type=\"loopback\" failed (delete): failed to find plugin \"loopback\" in path [/opt/cni/bin]"
Name: smartview-0-1710349133-778c969bf6-zvlvx
Namespace: default
Priority: 0
Service Account: smartview-0-1710349133
Node: red.portalify.com/10.80.9.125
Start Time: Wed, 13 Mar 2024 18:58:54 +0200
Labels: app.kubernetes.io/instance=smartview-0-1710349133
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=smartview
app.kubernetes.io/version=1.16.0
helm.sh/chart=smartview-0.1.0
pod-template-hash=778c969bf6
Annotations: <none>
Status: Terminating (lasts 58m)
Termination Grace Period: 30s
IP:
IPs: <none>
Controlled By: ReplicaSet/smartview-0-1710349133-778c969bf6
Containers:
smartview:
Container ID:
Image: nginx:1.16.0
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Liveness: http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tpfwx (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-tpfwx:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 103m default-scheduler Successfully assigned default/smartview-0-1710349133-778c969bf6-zvlvx to red.portalify.com
Warning FailedCreatePodSandBox 103m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "21b4f3641ba75912c6a3ee97081b1ce661468ea9d0b375749164f8702972e29d": plugin type="bridge" failed (add): failed to find plugin "bridge" in path [/opt/cni/bin]
Normal SandboxChanged 93m (x48 over 103m) kubelet Pod sandbox changed, it will be killed and re-created.
Normal SandboxChanged 73m (x2 over 73m) kubelet Pod sandbox changed, it will be killed and re-created.
Normal SandboxChanged 62m (x47 over 72m) kubelet Pod sandbox changed, it will be killed and re-created.
Warning FailedKillPod 57m (x7 over 59m) kubelet error killing pod: failed to "KillPodSandbox" for "aa558c30-21a8-4d17-ba9b-bb8a2852f79d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"21b4f3641ba75912c6a3ee97081b1ce661468ea9d0b375749164f8702972e29d\": plugin type=\"loopback\" failed (delete): failed to find plugin \"loopback\" in path [/opt/cni/bin]"
Warning FailedMount 50m (x9 over 52m) kubelet MountVolume.SetUp failed for volume "kube-api-access-tpfwx" : [failed to fetch token: serviceaccounts "smartview-0-1710349133" not found, object "default"/"kube-root-ca.crt" not registered]
Warning FailedKillPod 2m15s (x233 over 52m) kubelet error killing pod: failed to "KillPodSandbox" for "aa558c30-21a8-4d17-ba9b-bb8a2852f79d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"21b4f3641ba75912c6a3ee97081b1ce661468ea9d0b375749164f8702972e29d\": plugin type=\"loopback\" failed (delete): failed to find plugin \"loopback\" in path [/opt/cni/bin]"
[root@red ~]# kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cloud-controller-manager-red.portalify.com 1/1 Running 16 (55m ago) 8s 10.80.9.125 red.portalify.com <none> <none>
etcd-red.portalify.com 1/1 Running 0 54m 10.80.9.125 red.portalify.com <none> <none>
helm-install-rke2-canal-qhj7v 0/1 Completed 0 3h3m 10.80.9.125 red.portalify.com <none> <none>
helm-install-rke2-coredns-t28x7 0/1 Completed 0 3h3m 10.80.9.125 red.portalify.com <none> <none>
helm-install-rke2-ingress-nginx-5wl59 0/1 ContainerCreating 0 3h3m <none> red.portalify.com <none> <none>
helm-install-rke2-metrics-server-7jx6r 0/1 ContainerCreating 0 3h3m <none> red.portalify.com <none> <none>
helm-install-rke2-snapshot-controller-crd-nz2f4 0/1 ContainerCreating 0 3h3m <none> red.portalify.com <none> <none>
helm-install-rke2-snapshot-controller-drgvg 0/1 ContainerCreating 0 3h3m <none> red.portalify.com <none> <none>
helm-install-rke2-snapshot-validation-webhook-zq2c8 0/1 ContainerCreating 0 3h3m <none> red.portalify.com <none> <none>
kube-apiserver-red.portalify.com 1/1 Running 2 59m 10.80.9.125 red.portalify.com <none> <none>
kube-controller-manager-red.portalify.com 1/1 Running 17 (55m ago) 3h3m 10.80.9.125 red.portalify.com <none> <none>
kube-scheduler-red.portalify.com 1/1 Running 3 (55m ago) 3h3m 10.80.9.125 red.portalify.com <none> <none>
rke2-canal-qrvs7 0/2 Pending 0 176m <none> <none> red.portalify.com <none>
rke2-coredns-rke2-coredns-94745d-2tjx2 0/1 Pending 0 176m <none> <none> <none> <none>
rke2-coredns-rke2-coredns-autoscaler-d8587b89c-r7ms7 0/1 Pending 0 176m <none> <none> <none> <none> |
Beta Was this translation helpful? Give feedback.
Answered by
brandond
Mar 13, 2024
Replies: 2 comments 3 replies
-
Your CNI pod is stuck pending. You'd have to describe the pod to figure out why. Until the CNI comes up, there is no CNI installed on your node. Which is why the CNI directories are empty. |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
MattiL
-
|
rke2-canal-qrvs7 cannot be described because it is not found. [root@red ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default smartview-0-1710346441-bc78d4575-mskfp 0/1 Terminating 0 16h
default smartview-0-1710349133-778c969bf6-zvlvx 0/1 Terminating 0 16h
kube-system cloud-controller-manager-red.portalify.com 0/1 Pending 0 6s
kube-system etcd-red.portalify.com 1/1 Running 0 15h
kube-system helm-install-rke2-canal-qhj7v 0/1 Completed 0 17h
kube-system helm-install-rke2-coredns-t28x7 0/1 Completed 0 17h
kube-system helm-install-rke2-ingress-nginx-5wl59 0/1 ContainerCreating 0 17h
kube-system helm-install-rke2-metrics-server-7jx6r 0/1 ContainerCreating 0 17h
kube-system helm-install-rke2-snapshot-controller-crd-nz2f4 0/1 ContainerCreating 0 17h
kube-system helm-install-rke2-snapshot-controller-drgvg 0/1 ContainerCreating 0 17h
kube-system helm-install-rke2-snapshot-validation-webhook-zq2c8 0/1 ContainerCreating 0 17h
kube-system kube-apiserver-red.portalify.com 1/1 Running 2 15h
kube-system kube-controller-manager-red.portalify.com 1/1 Running 18 (7h38m ago) 7h37m
kube-system kube-scheduler-red.portalify.com 1/1 Running 3 (15h ago) 17h
kube-system rke2-canal-qrvs7 0/2 Pending 0 17h
kube-system rke2-coredns-rke2-coredns-94745d-2tjx2 0/1 Pending 0 17h
kube-system rke2-coredns-rke2-coredns-autoscaler-d8587b89c-r7ms7 0/1 Pending 0 17h[root@red ~]# /var/lib/rancher/rke2/bin/kubectl describe pods rke2-canal-qrvs7 |
Beta Was this translation helpful? Give feedback.
3 replies
-
You forgot to specify the namespace when describing: |
Beta Was this translation helpful? Give feedback.
-
[root@red ~]# kubectl describe pod -n kube-system rke2-canal-qrvs7
Name: rke2-canal-qrvs7
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Service Account: canal
Node: <none>
Labels: controller-revision-hash=79c77f676f
k8s-app=canal
pod-template-generation=1
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: DaemonSet/rke2-canal
NominatedNodeName: red.portalify.com
Init Containers:
install-cni:
Image: rancher/hardened-calico:v3.27.0-build20240206
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/install
Environment:
CALICO_CNI_SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
CNI_CONF_NAME: 10-canal.conflist
CNI_NETWORK_CONFIG: <set to the key 'cni_network_config' of config map 'rke2-canal-config'> Optional: false
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CNI_MTU: <set to the key 'veth_mtu' of config map 'rke2-canal-config'> Optional: false
SLEEP: false
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/host/opt/cni/bin from cni-bin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cr5z9 (ro)
flexvol-driver:
Image: rancher/hardened-calico:v3.27.0-build20240206
Port: <none>
Host Port: <none>
Command:
/usr/local/bin/flexvol.sh
-s
/usr/local/bin/flexvol
-i
flexvoldriver
Environment: <none>
Mounts:
/host/driver from flexvol-driver-host (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cr5z9 (ro)
Containers:
calico-node:
Image: rancher/hardened-calico:v3.27.0-build20240206
Port: <none>
Host Port: <none>
Command:
start_runit
Requests:
cpu: 250m
Liveness: exec [/usr/bin/calico-node -felix-live] delay=10s timeout=10s period=10s #success=1 #failure=6
Readiness: http-get http://localhost:9099/readiness delay=0s timeout=10s period=10s #success=1 #failure=3
Environment:
DATASTORE_TYPE: kubernetes
USE_POD_CIDR: true
WAIT_FOR_DATASTORE: true
NODENAME: (v1:spec.nodeName)
CALICO_CNI_SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
CALICO_NETWORKING_BACKEND: none
CLUSTER_TYPE: k8s,canal
FELIX_IPTABLESREFRESHINTERVAL: 60
FELIX_IPTABLESBACKEND: auto
CALICO_DISABLE_FILE_LOGGING: true
FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT
FELIX_IPV6SUPPORT: false
FELIX_LOGSEVERITYSCREEN: info
FELIX_HEALTHENABLED: true
FELIX_PROMETHEUSMETRICSENABLED: true
FELIX_XDPENABLED: false
FELIX_FAILSAFEINBOUNDHOSTPORTS:
FELIX_FAILSAFEOUTBOUNDHOSTPORTS:
IP_AUTODETECTION_METHOD: first-found
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/var/lib/calico from var-lib-calico (rw)
/var/log/calico/cni from cni-log-dir (ro)
/var/run/calico from var-run-calico (rw)
/var/run/nodeagent from policysync (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cr5z9 (ro)
kube-flannel:
Image: rancher/hardened-flannel:v0.24.2-build20240122
Port: <none>
Host Port: <none>
Command:
/opt/bin/flanneld
--ip-masq
--kube-subnet-mgr
Environment:
POD_NAME: rke2-canal-qrvs7 (v1:metadata.name)
POD_NAMESPACE: kube-system (v1:metadata.namespace)
FLANNELD_IFACE: <set to the key 'canal_iface' of config map 'rke2-canal-config'> Optional: false
FLANNELD_IFACE_REGEX: <set to the key 'canal_iface_regex' of config map 'rke2-canal-config'> Optional: false
FLANNELD_IP_MASQ: <set to the key 'masquerade' of config map 'rke2-canal-config'> Optional: false
Mounts:
/etc/kube-flannel/ from flannel-cfg (rw)
/run/xtables.lock from xtables-lock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cr5z9 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
var-run-calico:
Type: HostPath (bare host directory volume)
Path: /var/run/calico
HostPathType:
var-lib-calico:
Type: HostPath (bare host directory volume)
Path: /var/lib/calico
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
flannel-cfg:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rke2-canal-config
Optional: false
cni-bin-dir:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType:
cni-net-dir:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
cni-log-dir:
Type: HostPath (bare host directory volume)
Path: /var/log/calico/cni
HostPathType:
policysync:
Type: HostPath (bare host directory volume)
Path: /var/run/nodeagent
HostPathType: DirectoryOrCreate
flexvol-driver-host:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/volumeplugins/nodeagent~uds
HostPathType: DirectoryOrCreate
kube-api-access-cr5z9:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: :NoSchedule op=Exists
:NoExecute op=Exists
CriticalAddonsOnly op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 13m (x7845 over 24h) default-scheduler 0/1 nodes are available: 1 Insufficient cpu. |
Beta Was this translation helpful? Give feedback.
-
|
Well that's pretty cut and dried isn't it: your node does not meet the basic system requirements, and the kubelet cannot schedule any additional pods. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Your CNI pod is stuck pending. You'd have to describe the pod to figure out why. Until the CNI comes up, there is no CNI installed on your node. Which is why the CNI directories are empty.