[BUG] docker-compose build pulls image from registry instead of local

[logopk]

Description

I have built a local image before pushing to my registry (registry:5000/logopk/httpd:latest).
It contains a new apache module that I build in the Dockerfile.

Now I have a second image that builds from this image.

That build succeeds (pulling from registry:5000) but the container fails as the module can't be found (it is not in the registry yet).

Problem also exists if the registry is offline - the build fails as the image can't be pulled.

When I revert to 2.9.0 this works as expected.

Steps To Reproduce

see above

Compose Version

docker-compose 2.12.0 from homebrew

Docker Environment

Client: Docker Engine - Community
 Version:           20.10.20
 API version:       1.41
 Go version:        go1.19.2
 Git commit:        9fdeb9c3de
 Built:             Tue Oct 18 17:43:24 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.20
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       03df974
  Built:            Tue Oct 18 18:18:16 2022
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Anything else?

No response

[raysn0w]

Build from local using docker compose has been broken for a while now, did you find any workarounds? I'm using docker build in the meantime.

[babyhuey]

Also getting this issue on v2.12.2

[babyhuey]

Confirm this works in v2.10 but v2.11 it's broken

[raysn0w]

Workaround: DOCKER_BUILDKIT=0 docker compose build I wonder if this is a BUILDKIT issue rather than just docker compose.

[ndeloof]

buildkit indeed allows to run builds in parallel, but doesn't manage dependencies between builds when one service image relies on another service image to be built before and available in local image cache (docker/buildx#447).

Closing as a duplicate for #8538

[raysn0w]

Could someone reopen this? This is different from #8538 in my case. This recently started failing with docker compose, and I'm trying to build from a local image already present in my local registry.

[babyhuey]

Yea, this is definitely not a dupe of 8538, needs to be reopened and looked at

[ndeloof]

Could you maybe setup a minimal reproduction example to help us investigate this bug ?

[babyhuey]

Dockerfile.dev

FROM ubuntu
RUN apt update && apt upgrade -y 

Dockerfile

FROM ubuntu-updated
RUN apt install -y python3-pip && pip install yamllint

docker-compose.yml

---
version: '3'
services:
  web:
    build:
      context: .
    ports:
      - 8503:8503   

docker build -t ubuntu-updated -f Dockerfile.dev . to build the local file

docker compose buildto pull from the newly created image that is tagged with the FROM name. and you get the error:

❯ docker compose build                                                                                                                                                                                             ⏎
[+] Building 0.2s (4/4) FINISHED                                                                                                                                                                                     
 => [internal] load .dockerignore                                                                                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                                                                                 0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                            0.0s
 => => transferring dockerfile: 112B                                                                                                                                                                            0.0s
 => ERROR [internal] load metadata for docker.io/library/ubuntu-updated:latest                                                                                                                                  0.1s
 => [auth] library/ubuntu-updated:pull token for registry-1.docker.io                                                                                                                                           0.0s
------
 > [internal] load metadata for docker.io/library/ubuntu-updated:latest:
------
failed to solve: ubuntu-updated: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed                                          

[babyhuey]

Huh, looking at that I see buildx that I hadn't noticed before. If I remove that, it is working as expected...

[raysn0w]

Huh, looking at that I see buildx that I hadn't noticed before. If I remove that, it is working as expected...

Did you remove it by running DOCKER_BUILDKIT=0 docker compose build or something else?

[babyhuey]

Huh, looking at that I see buildx that I hadn't noticed before. If I remove that, it is working as expected...

Did you remove it by running DOCKER_BUILDKIT=0 docker compose build or something else?

I ran a few commands, not sure which did it
docker buildx uninstall
and mv ~/.docker/buildx ~

[ndeloof]

I can't reproduce this bug with provided example 🧐

 $ docker build -t ubuntu-updated -f Dockerfile.dev .
[+] Building 7.5s (6/6) FINISHED                                                                                             
 => [internal] load build definition from Dockerfile.dev                                                                0.0s
 => => transferring dockerfile: 87B                                                                                     0.0s
 => [internal] load .dockerignore                                                                                       0.0s
 => => transferring context: 2B                                                                                         0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                        0.0s
 => [1/2] FROM docker.io/library/ubuntu                                                                                 0.0s
 => [2/2] RUN apt update && apt upgrade -y                                                                              7.4s
 => exporting to image                                                                                                  0.1s
 => => exporting layers                                                                                                 0.1s
 => => writing image sha256:3f3b2370c87677408c23635fc1a08d8733d1cc03a2669b13b0f22fc36b65f7a8                            0.0s
 => => naming to docker.io/library/ubuntu-updated                                                                       0.0s
 
$ docker compose build                                                                                                   
[+] Building 41.3s (6/6) FINISHED                                                                                            
 => [internal] load build definition from Dockerfile                                                                    0.0s
 => => transferring dockerfile: 112B                                                                                    0.0s
 => [internal] load .dockerignore                                                                                       0.0s
 => => transferring context: 2B                                                                                         0.0s
 => [internal] load metadata for docker.io/library/ubuntu-updated:latest                                                0.0s
 => [1/2] FROM docker.io/library/ubuntu-updated                                                                         0.0s
 => [2/2] RUN apt install -y python3-pip && pip install yamllint                                                       40.5s
 => exporting to image                                                                                                  0.8s
 => => exporting layers                                                                                                 0.8s
 => => writing image sha256:0c7ce3f2bc5480822218f4c7b2f238d6320f5990a13dff935047ece4af57e854                            0.0s 
 => => naming to docker.io/library/truc-web                                                                             0.0s 
                                                                                                                             
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them                         
$ 

[ndeloof]

Could you please run docker buildx inspect to check the configuration for your active builder ?

[pipozzz]

I consider this as serious issue any ETA ?

[michaelmworthington]

I spent about 4 hours today struggling with this same thing.

Similar to babyhuey, I have a base image that I was making changes to in parallel with a child image. Given that they are separate projects, they are not in the same compose file.

I build the base image so it is in my Docker Desktop for Mac (M1) list of images.

If I docker build the child image it works. If I docker compose (v2) to build the child image it bypasses the local cache and always looks for the image in Docker Hub.

Setting the environment variables to disable Build Kit did not work for me. I reverted my Docker Desktop configuration to use Compose V1 and am able to successfully docker-compose (v1) to build the child image. Note that after this change, docker compose (v2) fails in the same way as before.

[SpecLad]

I hit this bug as well. I did a Git bisect and the first "bad" commit is 8b1b708.

I'll see if I can create a minimal reproduction recipe.

[SpecLad]

Okay, it's just this:

$ cat docker-compose.yml
services:
  bazquux:
    build:
      context: .
$ cat Dockerfile
FROM foobar
$ docker image pull busybox:musl
musl: Pulling from library/busybox
e96e863613a1: Pull complete
Digest: sha256:e7dc28a9c45363cb558fd4a03bc65a21b602a4fd744d48a4002790ea2c988178
Status: Downloaded newer image for busybox:musl
docker.io/library/busybox:musl
$ docker image tag busybox:musl foobar
$ docker buildx create --use
nifty_carver
$ docker compose build
[+] Building 4.5s (4/4) FINISHED
 => [internal] booting buildkit                                                                                        2.8s
 => => pulling image moby/buildkit:buildx-stable-1                                                                     1.7s
 => => creating container buildx_buildkit_nifty_carver0                                                                1.1s
 => [internal] load .dockerignore                                                                                      0.2s
 => => transferring context: 2B                                                                                        0.0s
 => [internal] load build definition from Dockerfile                                                                   0.2s
 => => transferring dockerfile: 49B                                                                                    0.0s
 => ERROR [internal] load metadata for docker.io/library/foobar:latest                                                 1.2s
------
 > [internal] load metadata for docker.io/library/foobar:latest:
------
failed to solve: foobar: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

The docker buildx create command is critical, without it everything works as expected.

EDIT: added some missing commands

[treus]

I hit this problem as well, whereas colleagues did not have the problem when building the same project.

I 'fixed' it by doing the following:

• stop any moby/buildkit:buildx-stable-1 container if it's running
• run docker buildx rm --all-inactive
• then build the base-image with docker compose build
  • I assume this creates the buildx docker driver by default, given that I ran docker buildx create --driver docker --use default before but it failed with the error ERROR: additional instances of driver "docker" cannot be created even though docker buildx inspect did not show any docker driver
• then build the image that uses the local base-image with docker compose build

Before the 'fix', docker buildx inspect showed (I already stopped the container)

Name:          confident_brattain
Driver:        docker-container
Last Activity: 2023-05-10 09:41:26 +0000 UTC

Nodes:
Name:      confident_brattain0
Endpoint:  unix:///var/run/docker.sock
Status:    inactive
Platforms: 

After the 'fix', docker buildx inspect shows

Name:          default
Driver:        docker
Last Activity: 2023-05-10 09:50:08 +0000 UTC

Nodes:
Name:      default
Endpoint:  default
Status:    running
Buildkit:  23.0.5
Platforms: linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

It seems the docker-container driver does not use the local image repository, whereas the docker driver does.
I don't know what causes this, or what determines whether docker-container or docker driver is used for buildx, but this is what I observed...

[ndeloof]

docker buildkit driver is the one directly integrated in docker engine, and as such use the local image store. Builders created with docker-container have their own image store and cache, which is exported to the container runtime after build completion (if configured to), which should be the case for a build triggered by docker compose build

[pokiz]

The same happened to me yesterday.
I was using a docker-container builder, and it failed to detect and use the local image from the cache, built on the same run.
Now back to the default builder and it works, local image is detected as expected, and not pulled from remote.
This should be a bug?

[ndeloof]

Use of a docker-container builder involved builder is "isolated" from docker engine and as such can't rely on previously pulled image or built image exported to engine.
I'm closing this issue as not related to Docker Compose

[logopk]

@ndeloof I don't disagree that this is not (!) compose-related - however i would like to see a more elaborate answer here as how to fix this problem!

I have no clue how and why this happens in my environment - let alone how to prevent it from happening (again)

[ndeloof]

@logopk better ask docker/buildx for recommendations on using a docker-container builder and caching built images. Docker Compose is just a third-party user of such features and I can't tell you about the best approach

[benmoss]

You may be running into the issue that the image in your local registry is a different architecture than your system (amd64 vs arm64 if you're on a recent Mac). Specifying the platform via platforms: ["linux/amd64"] fixed it for me.

[ap-wtioit]

Sticking with the example docker build . does the following:

[+] Building 0.3s (5/5) FINISHED                                                                                                                                          docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                0.0s
 => => transferring dockerfile: 49B                                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/foobar:latest                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                     0.0s
 => [1/1] FROM docker.io/library/foobar:latest                                                                                                                                      0.0s
 => exporting to image                                                                                                                                                              0.0s
 => => exporting layers                                                                                                                                                             0.0s
 => => writing image sha256:825a9e1e525abc26cf061764c871274c07848447d551af88f0753fd9db71433d                                                                                        0.0s

while docker buildx build . fails with:

[+] Building 0.9s (3/3) FINISHED                                                                                                                      docker-container:ecstatic_ishizaka
 => [internal] load build definition from Dockerfile                                                                                                                                0.0s
 => => transferring dockerfile: 49B                                                                                                                                                 0.0s
 => ERROR [internal] load metadata for docker.io/library/foobar:latest                                                                                                              0.8s
 => [auth] library/foobar:pull token for registry-1.docker.io                                                                                                                       0.0s
------
 > [internal] load metadata for docker.io/library/foobar:latest:
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:1
--------------------
   1 | >>> FROM foobar
   2 |     
--------------------
ERROR: failed to solve: foobar: failed to resolve source metadata for docker.io/library/foobar:latest: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Which matches this discusstion in buildx:
docker/buildx#2418

Edit: this comment worked for me