Fix failing workflows

[Jose-Matsuda]

Some relevant information in --> #500 (comment)

I don't know what caused this, it doesnt seem to be the most recent changes Bryan did as I attempted reverting them but no difference

[Jose-Matsuda]

Bryan thought it could be something to do with memory, but the sizes of the images hasn't changed in a significant way since it started failing. I built jupyterlab-cpu locally from the main branch and it was just 7.74 GB

[bryanpaget]

We are almost always able to build RStudio (which weighs in at 9.2GB) and sometimes we can build JupyterLab-CPU (which weighs in at 7.57GB). JupyterLab-Pytorch is ~18GB.

pagetbr@l-pagetbr-2:~/aaw-kubeflow-containers/output/rstudio$ docker images
REPOSITORY                     TAG                            IMAGE ID       CREATED         SIZE
rstudio                        latest                         1a7696064cd2   3 hours ago     9.2GB
jupyterlab-cpu                 latest                         1a0dfc4c6a1b   23 hours ago    7.57GB
jupyter/datascience-notebook   ed2908bbb62e                   3a31ab32404b   10 months ago   4.43GB

[bryanpaget]

I have created a JIRA ticket to track this issue:

• https://jirab.statcan.ca/browse/CLOUD-19934

And I have posted an issue on the Github community message board:

• https://github.com/orgs/community/discussions/63974

While I wait for the Github community to respond to my post, I have the following courses of action.

• Look into self-hosted Github Runners, perhaps look into hosting them on the AAW itself.
• Look into the StatCan Gitlab instance and see if I can mirror the aaw-kubeflow-container repo into Gitlab and use the Gitlab CI/CD instead of the Github Runner.

The following tickets are blocked by this issue:

• Merge 22.04 update (from previous sprint). #504
• Research R, Python, Anaconda Update/Optimization #496
• Remove set-output from Makefiles. #497

[bryanpaget]

@vexingly discovered the version of Docker that Github Actions uses was updated right at the same time our builds started failing.

Runner

For the successful builds before August 04th 2023, we have the following image version:

Runner Image
  Image: ubuntu-22.04
  Version: 20230724.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20230724.1/images/linux/Ubuntu2204-Readme.md

And when the builds started failing, we have a new version of the runner (still based on Ubuntu 22.04):

Image: ubuntu-22.04
  Version: 20230728.3.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20230728.3/images/linux/Ubuntu2204-Readme.md

Docker

For the older, successful runs:

Checking docker version
  /usr/bin/docker version --format '{{.Server.APIVersion}}'
  '1.41'
  Docker daemon API version: '1.41'
  /usr/bin/docker version --format '{{.Client.APIVersion}}'
  '1.41'
  Docker client API version: '1.41'

The newer, failed runs:

Checking docker version
  /usr/bin/docker version --format '{{.Server.APIVersion}}'
  '1.42'
  Docker daemon API version: '1.42'
  /usr/bin/docker version --format '{{.Client.APIVersion}}'
  '1.42'
  Docker client API version: '1.42'

This led us to investigate the changelog (https://docs.docker.com/engine/api/version-history/) and the epiphany came from

GET /_ping and HEAD /_ping now return Builder-Version by default. This header contains the default builder to use, and is a recommendation as advertised by the daemon. However, it is up to the client to choose which builder to use.
The default value on Linux is version “2” (BuildKit), but the daemon can be configured to recommend version “1” (classic Builder). Windows does not yet support BuildKit for native Windows images, and uses “1” (classic builder) as a default.
This change is not versioned, and affects all API versions if the daemon has this patch.

So we added DOCKER_BUILDKIT=0 to our docker build command in the Makefile to disable BuildKit,

DOCKER_BUILDKIT=0 docker build $(DARGS) --rm --force-rm -t $$IMAGE_NAME ./output/$(notdir $@)

And now our builds are succeeding.

[bryanpaget]

#496 will have to be done next sprint.