devcontainer build doesn't go well with multi-CPU architecture support enabled

[sanyer]

If multi-CPU architecture support is enabled, devcontainer build action fails if features are specified in devcontainers.json:

...
[999 ms] local container features stored at: /Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/node_modules/vscode-dev-containers/container-features
[1000 ms] Start: Run: tar --no-same-owner -x -f -
[1021 ms] Start: Run: docker build -t dev_container_feature_content_temp -f /var/folders/f5/rlxvlqks455c6n8y863jcd1c0000gn/T/vsch/container-features/0.241.3-1657892458442/Dockerfile.buildContent /var/folders/f5/rlxvlqks455c6n8y863jcd1c0000gn/T/vsch/container-features/0.241.3-1657892458442
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load 
[+] Building 0.2s (4/4) FINISHED
...

As you can see, there is even a warning which mentions that the image will reside in build cache only.

The fix is to add argument --load to this build command. This argument also doesn't harm while docker driver is being used.

[chrmarti]

You can use --push as an additional option to the devcontainer CLI to push the images and manifest to your registry.

I thought --load doesn't make sense for multi-arch builds because the local image cache can only load images for the local architecture. Is this not the case?

[sanyer]

@chrmarti Well, it's devcontainer's image dev_container_feature_content_temp and it has to exist locally for next step docker buildx build to succeed. Oh, I'm sorry, I think I didn't include the actual error.

Here it is:

[2821 ms] Start: Run: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /var/folders/f5/rlxvlqks455c6n8y863jcd1c0000gn/T/vsch/container-features/0.241.3-1657892458442/Dockerfile-with-features -t vsc-devcontainers-cli-4bff2a3afe4c7458e8cff5cb44521237 --build-arg VARIANT=16-bullseye --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp /Users/romanzhuzha/workspace/personal/devcontainers-cli/.devcontainer
[+] Building 1.9s (5/5) FINISHED                                                
 => [internal] load build definition from Dockerfile-with-features         0.1s
 => => transferring dockerfile: 1.25kB                                     0.1s
 => [internal] load .dockerignore                                          0.2s
 => => transferring context: 2B                                            0.0s
 => ERROR [internal] load metadata for docker.io/library/dev_container_fe  1.7s
 => [internal] load metadata for mcr.microsoft.com/vscode/devcontainers/t  0.6s
 => [auth] library/dev_container_feature_content_temp:pull token for regi  0.0s
------
 > [internal] load metadata for docker.io/library/dev_container_feature_content_temp:latest:
------
Dockerfile-with-features:22
--------------------
  20 |     
  21 |     
  22 | >>> FROM dev_container_feature_content_temp as dev_containers_feature_content_source
  23 |     
  24 |     FROM $_DEV_CONTAINERS_BASE_IMAGE
--------------------
error: failed to solve: dev_container_feature_content_temp: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
[5456 ms] Error: Command failed: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /var/folders/f5/rlxvlqks455c6n8y863jcd1c0000gn/T/vsch/container-features/0.241.3-1657892458442/Dockerfile-with-features -t vsc-devcontainers-cli-4bff2a3afe4c7458e8cff5cb44521237 --build-arg VARIANT=16-bullseye --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp /Users/romanzhuzha/workspace/personal/devcontainers-cli/.devcontainer
[5456 ms]     at GR (/Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:240:1436)
[5456 ms]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[5456 ms]     at async Hu (/Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:239:2796)
[5456 ms]     at async tb (/Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:239:1779)
[5456 ms]     at async QR (/Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:245:2006)
[5456 ms]     at async cb (/Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:245:3110)
[5457 ms]     at async i_ (/Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:245:13194)
[5457 ms]     at async n_ (/Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:245:12950)
[5465 ms] Exit code 1
[5466 ms] Command failed: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper /Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/romanzhuzha/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --workspace-folder /Users/romanzhuzha/workspace/personal/devcontainers-cli --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/romanzhuzha/workspace/personal/devcontainers-cli --log-level debug --log-format json --config /Users/romanzhuzha/workspace/personal/devcontainers-cli/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --remove-existing-container --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[5466 ms] Exit code 1

All this is the output of Remote - Containers extension, btw.

[sanyer]

I'll properly report the issue.

Description

build command fails if multi-CPU architecture support is enabled and features are used. Switching back to default docker driver resolves the issue.

Steps to reproduce the issue:

1. Enable multi-CPU architecture support.

2. Try to build a devcontainer with features enabled.

Actual result:

Build fails with error message:

error: failed to solve: dev_container_feature_content_temp: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Expected result:

Build succeeds.

Remote - Containers version: v0.241.3

Additional info:

It happens as docker build in multi-CPU arch mode saves build result to cache instead of local or remote repo. The most straightforward fix would be adding argument --load for the docker build command which build temporary features image.

Full log:

...
[999 ms] local container features stored at: /Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/node_modules/vscode-dev-containers/container-features
[1000 ms] Start: Run: tar --no-same-owner -x -f -
[1021 ms] Start: Run: docker build -t dev_container_feature_content_temp -f /var/folders/f5/rlxvlqks455c6n8y863jcd1c0000gn/T/vsch/container-features/0.241.3-1657892458442/Dockerfile.buildContent /var/folders/f5/rlxvlqks455c6n8y863jcd1c0000gn/T/vsch/container-features/0.241.3-1657892458442
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load 
[+] Building 0.2s (4/4) FINISHED
...
[2821 ms] Start: Run: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /var/folders/f5/rlxvlqks455c6n8y863jcd1c0000gn/T/vsch/container-features/0.241.3-1657892458442/Dockerfile-with-features -t vsc-devcontainers-cli-4bff2a3afe4c7458e8cff5cb44521237 --build-arg VARIANT=16-bullseye --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp /Users/romanzhuzha/workspace/personal/devcontainers-cli/.devcontainer
[+] Building 1.9s (5/5) FINISHED                                                
 => [internal] load build definition from Dockerfile-with-features         0.1s
 => => transferring dockerfile: 1.25kB                                     0.1s
 => [internal] load .dockerignore                                          0.2s
 => => transferring context: 2B                                            0.0s
 => ERROR [internal] load metadata for docker.io/library/dev_container_fe  1.7s
 => [internal] load metadata for mcr.microsoft.com/vscode/devcontainers/t  0.6s
 => [auth] library/dev_container_feature_content_temp:pull token for regi  0.0s
------
 > [internal] load metadata for docker.io/library/dev_container_feature_content_temp:latest:
------
Dockerfile-with-features:22
--------------------
  20 |     
  21 |     
  22 | >>> FROM dev_container_feature_content_temp as dev_containers_feature_content_source
  23 |     
  24 |     FROM $_DEV_CONTAINERS_BASE_IMAGE
--------------------
error: failed to solve: dev_container_feature_content_temp: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
[5456 ms] Error: Command failed: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /var/folders/f5/rlxvlqks455c6n8y863jcd1c0000gn/T/vsch/container-features/0.241.3-1657892458442/Dockerfile-with-features -t vsc-devcontainers-cli-4bff2a3afe4c7458e8cff5cb44521237 --build-arg VARIANT=16-bullseye --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp /Users/romanzhuzha/workspace/personal/devcontainers-cli/.devcontainer
[5456 ms]     at GR (/Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:240:1436)
[5456 ms]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[5456 ms]     at async Hu (/Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:239:2796)
[5456 ms]     at async tb (/Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:239:1779)
[5456 ms]     at async QR (/Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:245:2006)
[5456 ms]     at async cb (/Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:245:3110)
[5457 ms]     at async i_ (/Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:245:13194)
[5457 ms]     at async n_ (/Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:245:12950)
[5465 ms] Exit code 1
[5466 ms] Command failed: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper /Users/romanzhuzha/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/romanzhuzha/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --workspace-folder /Users/romanzhuzha/workspace/personal/devcontainers-cli --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/romanzhuzha/workspace/personal/devcontainers-cli --log-level debug --log-format json --config /Users/romanzhuzha/workspace/personal/devcontainers-cli/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --remove-existing-container --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[5466 ms] Exit code 1
...

[chrmarti]

What do you get for docker buildx version? (This seems to be #60 with buildx.)

[sanyer]

$ docker buildx version
github.com/docker/buildx v0.7.1 05846896d149da05f3d6fd1e7770da187b52a247

[sanyer]

I had buildx manually installed. Deleting it switched to Docker Desktop bundled-in version github.com/docker/buildx v0.8.2 6224def4dd2c3d347eee19db595348c50d7cb491.
This seems to fix the issue.

[chrmarti]

Looks like microsoft/vscode-remote-release#7305. Closing for now. Thanks.