failed to destroy network for sandbox \"e2aa....... cni plugin not initialized

[eliassal]

What happened:

I am trying to reset a k8S to do kubadm init but it is failing with error

cni not initialized

I te seems that there are stuck pods and when I run
sudo crictl -r /var/run/containerd/containerd.sock pods

I get a bunch of pds in not ready state

I tried to remove them with
sudo crictl -r /var/run/containerd/containerd.sock rmp pod 18b112e4c2a2c
or
sudo crictl -r /var/run/containerd/containerd.sock pods |grep NotReady|cut -f1 -d" "|xargs -L 1 -I {} -t sudo crictl -r /var/run/containerd/containerd.sock rmp {}

I always get

xargs: warning: options --max-lines and --replace/-I/-i are mutually exclusive, ignoring previous --max-lines value
sudo crictl -r /var/run/containerd/containerd.sock rmp e2aa1be576def
E1120 19:36:30.972427    8311 log.go:32] "RemovePodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to forcibly stop sandbox \"e2aa1be576def963bb199d93203b234e4fb4df279fdbbe6fa7fe915c33e41ac1\": failed to destroy network for sandbox \"e2aa1be576def963bb199d93203b234e4fb4df279fdbbe6fa7fe915c33e41ac1\": cni plugin not initialized" podSandboxID="e2aa1be576def"
removing the pod sandbox "e2aa1be576def": rpc error: code = Unknown desc = failed to forcibly stop sandbox "e2aa1be576def963bb199d93203b234e4fb4df279fdbbe6fa7fe915c33e41ac1": failed to destroy network for sandbox "e2aa1be576def963bb199d93203b234e4fb4df279fdbbe6fa7fe915c33e41ac1": cni plugin not initialized
sudo crictl -r /var/run/containerd/containerd.sock rmp 18b112e4c2a2c
E1120 19:36:31.020208    8320 log.go:32] "RemovePodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to forcibly stop sandbox \"18b112e4c2a2c92ad682afa3ba65a24c0e9a3fc104156874990150b344c54056\": failed to destroy network for sandbox \"18b112e4c2a2c92ad682afa3ba65a24c0e9a3fc104156874990150b344c54056\": cni plugin not initialized" podSandboxID="18b112e4c2a2c"
removing the pod sandbox "18b112e4c2a2c": rpc error: code = Unknown desc = failed to forcibly stop sandbox "18b112e4c2a2c92ad682afa3ba65a24c0e9a3fc104156874990150b344c54056": failed to destroy network for sandbox "18b112e4c2a2c92ad682afa3ba65a24c0e9a3fc104156874990150b344c54056": cni plugin not initialized
sudo crictl -r /var/run/containerd/containerd.sock rmp 8141c56986d0b
E1120 19:36:31.066335    8328 log.go:32] "RemovePodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to forcibly stop sandbox \"8141c56986d0b6f17187855c02c059cc7a2356dad7eda6343f34015e833cffc2\": failed to destroy network for sandbox \"8141c56986d0b6f17187855c02c059cc7a2356dad7eda6343f34015e833cffc2\": cni plugin not initialized" podSandboxID="8141c56986d0b"
removing the pod sandbox "8141c56986d0b": rpc error: code = Unknown desc = failed to forcibly stop sandbox "8141c56986d0b6f17187855c02c059cc7a2356dad7eda6343f34015e833cffc2": failed to destroy network for sandbox "8141c56986d0b6f17187855c02c059cc7a2356dad7eda6343f34015e833cffc2": cni plugin not initialized
sudo crictl -r /var/run/containerd/containerd.sock rmp f58d583475174
E1120 19:36:31.111799    8336 log.go:32] "RemovePodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to forcibly stop sandbox \"f58d58347517496973554f1d58f4b536770121f23a204b1e6c40120abb68cc5d\": failed to destroy network for sandbox \"f58d58347517496973554f1d58f4b536770121f23a204b1e6c40120abb68cc5d\": cni plugin not initialized" podSandboxID="f58d583475174"
removing the pod sandbox "f58d583475174": rpc error: code = Unknown desc = failed to forcibly stop sandbox "f58d58347517496973554f1d58f4b536770121f23a204b1e6c40120abb68cc5d": failed to destroy network for sandbox "f58d58347517496973554f1d58f4b536770121f23a204b1e6c40120abb68cc5d": cni plugin not initialized
sudo crictl -r /var/run/containerd/containerd.sock rmp c1efa5e75889c

What you expected to happen:

Be able to remove those pods that does not allow kubeadm reset or init goes through

How to reproduce it (as minimally and precisely as possible):

Run kubadem reset

Anything else we need to know?:

Environment:

• Container runtime or hardware configuration: ubuntu 24.04
• OS (e.g: cat /etc/os-release):
• PRETTY_NAME="Ubuntu 22.04.5 LTS"
  NAME="Ubuntu"
  VERSION_ID="22.04"
  VERSION="22.04.5 LTS (Jammy Jellyfish)"
  VERSION_CODENAME=jammy
  ID=ubuntu
  ID_LIKE=debian
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  UBUNTU_CODENAME=jammy

[SergeyKanzhelev]

@eliassal this looks like the container runtime issue or misconfiguration. You would need to file a bug with the container runtime that you are using instead.

If you believe the issue is with the crictl, please reopen this bug and explain why you think that CRI tools is a problem.

/close

[k8s-ci-robot]

@SergeyKanzhelev: Closing this issue.

In response to this:

@eliassal this looks like the container runtime issue or misconfiguration. You would need to file a bug with the container runtime that you are using instead.

If you believe the issue is with the crictl, please reopen this bug and explain why you think that CRI tools is a problem.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.