GitHub Action Runners hang starting Docker sometimes: how to debug and fix?

[Nuru]

I'm running the current GitHub Action Runner (v2.317.0) on the current Bottlerocket (Bottlerocket OS 1.20.2 (aws-k8s-1.29)), with Docker-in-Docker.

Often it works, although there are some warnings about networking

Click for details

2024-06-25 13:10:52.291  DEBUG --- Docker enabled runner detected and Docker daemon wait is enabled
2024-06-25 13:10:52.293  DEBUG --- Waiting until Docker is available or the timeout of 120 seconds is reached
time="2024-06-25T13:11:37.487499789Z" level=warning msg="WARNING: bridge-nf-call-iptables is disabled"
time="2024-06-25T13:11:37.487538039Z" level=warning msg="WARNING: bridge-nf-call-ip6tables is disabled"
time="2024-06-25T13:11:37.487555249Z" level=info msg="Docker daemon" commit=311b9ff graphdriver=overlay2 version=24.0.7
time="2024-06-25T13:11:37.487844161Z" level=info msg="Daemon has completed initialization"
time="2024-06-25T13:11:37.541730398Z" level=info msg="API listen on /var/run/docker.sock"
}CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
2024-06-25 13:11:37.546  NOTICE --- WARNING LATEST TAG HAS BEEN DEPRECATED. SEE GITHUB ISSUE FOR DETAILS:
2024-06-25 13:11:37.548  NOTICE --- https://github.com/actions/actions-runner-controller/issues/2056

√ Connected to GitHub

Current runner version: '2.317.0'
2024-06-25 13:11:40Z: Listening for Jobs

It can take less than a second for Docker to start. In the example above it took 45 seconds.
However, often it fails (times out after 120 seconds), and I just can't figure out how to debug this. When it is in a failure mode, I cannot exec into the pod via kubectl exec, and I cannot even launch the admin container (it times out after 60 seconds). There is no other logging from the pod. I have poked around looking for logs, but have not found anything relevant, but then again, I'm not familiar with Bottlerocket, so I don't really know where to look.

I am allocating 2250m of CPU (over 2 full CPUs) and 6GiB of memory to the pod, so if it is a resource issue, it would have to be a system resource issue. Does running Docker in Docker from a privileged Pod put pressure on the system? Given that the Pod is just sleeping, that should leave 2 CPUs available for the system to get Docker running.

I suppose I should point out that I'm launching 90 pods all at once onto a r7i.48xlarge (192 vCPUs and 1536 GiB of RAM), and it looks like Bottlerocket is not reserving a proportionate amount of CPU and memory for kubelet and the rest of the system, but that does not feel like the issue to me. I mean, we launch 90 pods at once. They all time out after 120s (more like 140 seconds with setup), quit, and then we launch another 90 to replace them. I would think the second time around everything that needs to be set up (e.g. any kind of resource that is initialized on demand) would be set up in the first round and the second set of Pods would succeed. Once we get 90 pods running all at once, then the Docker startup goes back to under 1 second, but as long as we launch in waves, it all hangs up.

I have worked around the issue for now by extending the timeout from 120 seconds to 300 seconds, but that is kind of ridiculous. I would really like to find out what the issue is and get it fixed. I'm asking here because this is all running on Bottlerocket and I don't know where to look. I'm pretty disoriented compared to Amazon Linux.

[Nuru]

It appears this is due to disk I/O saturation, due to 50 Pods launching at once on this 196 vCPU instance, backed by a single normal gp3 EBS volume, and not a problem specific to Bottlerocket. My apologies.

[larvacea]

Thank you just the same. Especially thank you for sharing the root cause you found.