Not able to build Docker images. Starved for CPU/Memory?

[bryanpaget]

Hello Github Actions community,

I am getting the following error when building Docker images:

The hosted runner: GitHub Actions # lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

The repository in question is StatCan/aaw-kubeflow-containers and one of the most recent Actions can be found here.

Just wondering about possible reasons for this since we were able to build these images up to and including August 02 2023. Possible courses of investigation would be appreciated.

[bryanpaget]

I just made a mini build, where I stripped out most of the content for the Dockerfiles, and most of the images (including jupyterlab-pytorch) built! The logs show failures due to the images failing tests they can't possibly pass now that they are slimmed down. But this seems to support my hypothesis that something changed in the Github Actions runner.

https://github.com/StatCan/aaw-kubeflow-containers/actions/runs/5891896840/job/15980006378

[bryanpaget]

It has been discovered that the version of Docker used by the Github runner has changed from 1.41 to 1.42 during the timeframe when I noticed our images were failing to build.

[bryanpaget]

We have discovered the version of Docker that Github Actions uses was updated right at the same time our builds started failing.

Runner

For the successful builds before August 04th 2023, we have the following image version:

Runner Image
  Image: ubuntu-22.04
  Version: 20230724.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20230724.1/images/linux/Ubuntu2204-Readme.md

And when the builds started failing, we have a new version of the runner (still based on Ubuntu 22.04):

Image: ubuntu-22.04
  Version: 20230728.3.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20230728.3/images/linux/Ubuntu2204-Readme.md

Docker

For the older, successful runs:

Checking docker version
  /usr/bin/docker version --format '{{.Server.APIVersion}}'
  '1.41'
  Docker daemon API version: '1.41'
  /usr/bin/docker version --format '{{.Client.APIVersion}}'
  '1.41'
  Docker client API version: '1.41'

The newer, failed runs:

Checking docker version
  /usr/bin/docker version --format '{{.Server.APIVersion}}'
  '1.42'
  Docker daemon API version: '1.42'
  /usr/bin/docker version --format '{{.Client.APIVersion}}'
  '1.42'
  Docker client API version: '1.42'

This led us to investigate the changelog (https://docs.docker.com/engine/api/version-history/) and the epiphany came from

GET /_ping and HEAD /_ping now return Builder-Version by default. This header contains the default builder to use, and is a recommendation as advertised by the daemon. However, it is up to the client to choose which builder to use.
The default value on Linux is version “2” (BuildKit), but the daemon can be configured to recommend version “1” (classic Builder). Windows does not yet support BuildKit for native Windows images, and uses “1” (classic builder) as a default.
This change is not versioned, and affects all API versions if the daemon has this patch.

So we added DOCKER_BUILDKIT=0 to our docker build command in the Makefile to disable BuildKit,

DOCKER_BUILDKIT=0 docker build $(DARGS) --rm --force-rm -t $$IMAGE_NAME ./output/$(notdir $@)

And now our builds are succeeding.