chore: speedup docker build time

[jeluard]

Motivation

Speedup docker builds

Description

Tweak current Docker build process to improve build speeds.

This is achieved by:

• using regular node images for build layers; this remove the need to add extra tooling via apk every run
• document how to use yarn offline mirror to reduce the need to download dependencies

Using regular node images comes at a higher download cost (image is bigger) but is cached. slim images do not ship with the necessary tooling.

Note that the current Docker build creates a multi-platform Docker image (arm64 and amd64). We can probably improve things further by having a dedicated Dockerfile for dev usage.

Results

Before

1st run: 5m 22s
Subsequent runs, after code change: 4m 52s

After

1st run: 3m 16s
Subsequent runs, after code change: 2m 32s

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 62.20%. Comparing base (49c1689) to head (6da58fd).

Additional details and impacted files

@@            Coverage Diff            @@
##           unstable    #6787   +/-   ##
=========================================
  Coverage     62.19%   62.20%           
=========================================
  Files           571      571           
  Lines         60021    60021           
  Branches       1973     1976    +3     
=========================================
+ Hits          37333    37334    +1     
+ Misses        22645    22644    -1     
  Partials         43       43           

[nflaig]

need to confirm this does not increase the image size

[jeluard]

I added some experiment using a lodestar base builder image. This reduce the need to either use a big image or update via apk every build.
This image will have to be pushed to ChainSafe dockerhub beforehand.

[nflaig]

What was the argument against using the node:20 as builder image?

[jeluard]

For some unclear reason it fails to build native libs in the second stage. Might be the original reason for using alpine.
Also this one would be smaller and offers more opportunity for further tweaks down the road.

[nflaig]

I am really not sure it's worth to have a separate builder image just for these dependencies as the layer will always be retrieved from local cache after the first local build, and even if it is executed it just took 11 seconds on my machine.

The issues I see with using a separate builder image is that right now we automatically use the latest 20.x version which is good to get security updates in. There is also extra maintenance to keep this up-to-date if it's not part of the CI. Although we could build this image before running the main docker build.

[jeluard]

Moved back to a single node image.

[nflaig]

What if someone has settings in their global yarn config? can't we just commit a .yarnrc to the project with this setting in it?

[jeluard]

If we do that, the offline mirror will always be enabled. Not sure if we want that by default.

[nflaig]

what's the disadvantage other than storage increase which is not a lot.

> du -sh .yarn
396M    .yarn

The advice to move the global yarn config into the current folder is not ideal, what if there are other settings in this file, means the global yarn config was effectively deleted.

If we wanna support offline mirror would just add a .yarnrc in git

yarn-offline-mirror ".yarn/yarn-offline-mirror"
yarn-offline-mirror-pruning true

[nflaig]

Note that the current Docker build creates a multi-platform Docker image (arm64 and amd64). We can probably improve things further by having a dedicated Dockerfile for dev usage.

I don't think this is correct. By default docker will only create an image for the platform it's built on. To create multiarch images you need to pass --platform linux/amd64,linux/arm64 and requires an emulator like QEMU to be installed

[jeluard]

@nflaig Yes that's what we are doing, see this. If we were not doing multi-platform build the Dockerfile could be simplified to have only a single build stage (w/o the FROM --platform).

[nflaig]

@nflaig Yes that's what we are doing, see this. If we were not doing multi-platform build the Dockerfile could be simplified to have only a single build stage (w/o the FROM --platform).

ah you mean we could remove the middle stage in case we don't need to support arm64 images?

[jeluard]

In a separate Dockerfile only used for local dev, yes

[nflaig]

using regular node images for build layers; this remove the need to add extra tooling via apk every run

this layer will be retrieved from cache after the first run, but rather than reducing build time, a good argument for using the node:22 full image as base is that we reduce the likelyhood that systems deps are missing.

But overall, this might increase the build time in the CI as the node:22 image is not cached there, and afaik quite large to download.

[jeluard]

I would expect that if download time is an issue for CI, there would be ways to improve this via some caching.
The main goal of those changes is to speedup docker builds during dev cycles.

[nflaig]

The main goal of those changes is to speedup docker builds during dev cycles.

it's not clear to me then why we need to use a full node:22 as based on my observations the apk install layer will always be retrieved from cache after first run

[jeluard]

Hmm can't reproduce what I believe I was seeing during the interop indeed.

[nflaig]

did some testings with offline mirror vs. without, the build time seems to be roughly the same. The reason I would assume for this is that I have anyways all packages cached so there shouldn't be a difference.

Hmm can't reproduce what I believe I was seeing during the interop indeed.

maybe it still does some checks against the yarn registry? There were definitely request timeouts I noticed during the interop which we might be able to avoid by using a offline mirror.

[jeluard]

Doesn't appear to provide value.