K3s GPU Agent Node has all pods constantly restart

[Sharpz7]

Environmental Info:
K3s Version:

adammc@isaic:~$ k3s -v
k3s version v1.27.4+k3s1 (36645e73)
go version go1.20.6

Node(s) CPU architecture, OS, and Version:

adammc@isaic:~$ uname -a
Linux isaic.mcaq.me 5.15.0-83-generic #92-Ubuntu SMP Mon Aug 14 09:30:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

1 Server, 1 GPU Agent, configured using https://github.com/alexellis/k3sup that are on different clouds

Describe the bug:

Essentially, I cannot run anything on the agent node because after 1-2mins, the pods restart.

Steps To Reproduce:
I doubt this is re-produceable, and instead think its caused by something being configured differently where I am hosting the GPU agent node (On Openstack through https://isaic.ca/), but litrally just running k3sup join on my system creates this problem.

Expected behavior:

Default Pods (Node Exporter and Traefik) should come up and run fine

Actual behavior:

Pods constantly restart. The Traefik Pods have an error, the node exporter does not.

+ '[' '!' -e /pause ]
+ mkfifo /pause
/usr/bin/entry: line 84: can't open /pause: Interrupted system call

Additional context / logs:

adam@vmi1447591:~$ kubectl describe node isaic.mcaq.me
Name:               isaic.mcaq.me
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=isaic.mcaq.me
                    kubernetes.io/os=linux
                    node.kubernetes.io/instance-type=k3s
Annotations:        flannel.alpha.coreos.com/backend-data: {"PublicKey":"wxseHSBRKRJfuJYDL4eN4IZUZHurSJ9L3WRPbTZAkEk="}
                    flannel.alpha.coreos.com/backend-type: wireguard
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.168.1.56
                    k3s.io/hostname: isaic.mcaq.me
                    k3s.io/internal-ip: 192.168.1.56
                    k3s.io/node-args: ["agent"]
                    k3s.io/node-config-hash: FDWWD5WXW4EHUAZ3J63RGNVLRP6JN4DAMGK52EUDOPFYEN3XN6XQ====
                    k3s.io/node-env:
                      {"K3S_DATA_DIR":"/var/lib/rancher/k3s/data/dc43f496a0a9ac19d3b2444d390db38e0cfb38e672721f838b075422b8734994","K3S_TOKEN":"********","K3S_U...
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 07 Sep 2023 13:09:36 -0700
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  isaic.mcaq.me
  AcquireTime:     <unset>
  RenewTime:       Thu, 07 Sep 2023 13:32:05 -0700
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 07 Sep 2023 13:30:03 -0700   Thu, 07 Sep 2023 13:09:36 -0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 07 Sep 2023 13:30:03 -0700   Thu, 07 Sep 2023 13:09:36 -0700   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 07 Sep 2023 13:30:03 -0700   Thu, 07 Sep 2023 13:09:36 -0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 07 Sep 2023 13:30:03 -0700   Thu, 07 Sep 2023 13:09:36 -0700   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  192.168.1.56
  Hostname:    isaic.mcaq.me
Capacity:
  cpu:                16
  ephemeral-storage:  1016057248Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             64293428Ki
  pods:               110
Allocatable:
  cpu:                16
  ephemeral-storage:  988420490080
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             64293428Ki
  pods:               110
System Info:
  Machine ID:                 64f675d95c8d411fadf4c3e37c78b08c
  System UUID:                1bf3eda3-a208-46af-98c1-8b6fbd7233b8
  Boot ID:                    f38f6db9-40eb-4cb7-a2fe-d104d2e11ba2
  Kernel Version:             5.15.0-83-generic
  OS Image:                   Ubuntu 22.04.3 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.1-k3s1
  Kubelet Version:            v1.27.4+k3s1
  Kube-Proxy Version:         v1.27.4+k3s1
PodCIDR:                      10.42.8.0/24
PodCIDRs:                     10.42.8.0/24
ProviderID:                   k3s://isaic.mcaq.me
Non-terminated Pods:          (2 in total)
  Namespace                   Name                            CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                            ------------  ----------  ---------------  -------------  ---
  kube-system                 svclb-traefik-49c11ff9-p76gc    0 (0%)        0 (0%)      0 (0%)           0 (0%)         10m
  lens-metrics                node-exporter-ndk8c             10m (0%)      200m (1%)   24Mi (0%)        100Mi (0%)     10m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                10m (0%)   200m (1%)
  memory             24Mi (0%)  100Mi (0%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-1Gi      0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)

Some things in here appear strange to me. flannel.alpha.coreos.com/public-ip: 192.168.1.56 is set to the public IP, and from Lens https://k8slens.dev/ the memory is marked as N/A https://photos.app.goo.gl/xudHwja6YXhLZZ5r7

I have verified that the container runtime is okay

sudo grep nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml
sudo ctr image pull docker.io/nvidia/cuda:11.6.2-base-ubuntu20.04
sudo ctr run --rm -t --runc-binary=/usr/bin/nvidia-container-runtime --env NVIDIA_VISIBLE_DEVICES=all docker.io/nvidia/cuda:11.6.2-base-ubuntu20.04 cuda-11.6.2-base-ubuntu20.04 nvidia-smi

runtime = "nvidia-container-runtime"
adammc@isaic:~$ sudo ctr run --rm -t --runc-binary=/usr/bin/nvidia-container-runtime --env NVIDIA_VISIBLE_DEVICES=all docker.io/nvidia/cuda:11.6.2-base-ubuntu20.04 cuda-11.6.2-base-ubuntu20.04 nvidia-smi
Thu Sep  7 20:35:39 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.05              Driver Version: 535.86.05    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100 80GB PCIe          On  | 00000000:00:06.0 Off |                   On |
| N/A   31C    P0              41W / 300W |      0MiB / 81920MiB |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| MIG devices:                                                                          |
+------------------+--------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |                   Memory-Usage |        Vol|      Shared           |
|      ID  ID  Dev |                     BAR1-Usage | SM     Unc| CE ENC DEC OFA JPG    |
|                  |                                |        ECC|                       |
|==================+================================+===========+=======================|
|  No MIG devices found                                                                 |
+---------------------------------------------------------------------------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

Maybe one interesting thing to note, is other times I have configured k3s, and on the master node in this case, the internal node IP is always set to the external node IP, see

NAME                           STATUS   ROLES                       AGE   VERSION        INTERNAL-IP       EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
isaic.mcaq.me                  Ready    <none>                      29m   v1.27.4+k3s1   192.168.1.56      <none>        Ubuntu 22.04.3 LTS   5.15.0-83-generic   containerd://1.7.1-k3s1
vmi1447591.contaboserver.net   Ready    control-plane,etcd,master   2d    v1.27.4+k3s1   144.126.146.150   <none>        Ubuntu 20.04.6 LTS   5.4.0-162-generic   containerd://1.7.1-k3s1

The contabo nodes internal IP is actually its external IP. I wonder if that has anything to do with this?

I also got a cgroup systemd error originally when setting this up, so every time I reboot I also run

sudo mkdir /sys/fs/cgroup/systemd
sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd

Also, legacy iptables were not originally installed, so I fixed that with

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

Again, not sure if that info is relevant but thought it would be useful.

[brandond]

I cannot run anything on the agent node because after 1-2mins, the pods restart.

• This looks like a duplicate of pods with runtimeClassName: nvidia get killed after a few seconds #7130.

I'm not sure that it was ever resolved; but it's not something that K3s or containerd would do. Best I can tell something in the GPU operator is terminating the pods, but I have no idea why it would do that.

[Sharpz7]

Hey, so I don't think it is, as its all pods on the node, even the ones without the runtime option.

[Sharpz7]

@brandond any other suggestions?

[caroline-suse-rancher]

Converting to a discussion, since this doesn't seem to be a k3s bug