docker-compose using buildkit fails to fetch local images (already existing or build by another service)

[typoworx-de]

docker-compose using buildkit fails to fetch local images (already existing or build by another service).

This bug obviously has been discussed here:
docker/compose#3660

I'm still having the issue and found this thread! I don't think disabling Buildkit is a smart hack if one wants to use buildkit-features like COPY --chmod=a+x

The matter that buildkit obviously breaks this is a bug. Has this been reported to buildkit or is the bug located in docker itself?

example:

services:
  build-app:
     image: my-app-base
      context: .
      dockerfile: ./Dockerfile.base
      target: base-image

    run-app:
      image: my-app-project
      build:
        context: .
        dockerfile: ./Dockerfile.project
        target: base-image

FROM my-app-base AS my-project-app

---

Update
A simple reproducer:
https://github.com/Gabriel-Kaufmann/reproducer-docker-compose-buildx-issue

[filinvadim]

For latest Docker versions that happens because you haven't pointed base image platform type like linux/amd64 for example.

[crazy-max]

@typoworx-de

docker-compose using buildkit fails to fetch local images

Can you post logs please? Not sure what service fails

Also do you repro with docker build command?:

$ docker build --target base-image -f ./Dockerfile.base -t my-app-base .
$ docker build --target base-image -f ./Dockerfile.project -t my-app-project .

[typoworx-de]

@crazy-max, @filinvadim thanks for your help.

I've trying to build without docker-compose using "plain" docker with buildx:
docker build --target stage-release -f .docker/services/httpd-php/Dockerfile.apache2-modphp --load .
(don't wonder my path/compose setup is slightly different than in example)

Error from running the docker build:

 > [stage-release 4/6] COPY --chmod=a+x ./rootfs/docker-entrypoint.sh /docker-entrypoint.sh:
------
Dockerfile.apache2-modphp:199
--------------------
 197 |     
 198 |     COPY ./rootfs/docker-entrypoint.d /docker-entrypoint.d
 199 | >>> COPY --chmod=a+x ./rootfs/docker-entrypoint.sh /docker-entrypoint.sh

ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref y4xwnno4wn858ct0my20dfwaz::sjf5a32z3kkjfzazoceot6y2i: "/rootfs/docker-entrypoint.sh": not found

I'm running the build from project-root where docker-compose.yml is, which has extended the docker-compose.build.yml. I think the problem here is, which is the referencing path (or actually the build-context) that is being used? is it the build.context "." where docker-compose.build.yml is located whereas "." will be the Path of that file? Or will I have to use path relative to project-root where the build is started from? This is somehow confusing while looking on "context: ." setting in view of docker-compose.yml and build-options. I would expect that "context: {path}" is the mandatory part that tells what dockerfile will be using as reference?!

is context: . relative to the current docker-compose (extend) file or is the path relative to the path of project-root docker-compose.yml?

May be you can help me to deconfuse that :)

I've been testing these context's in
context: .docker/services/httpd-php/
and
context: .

both failing with same error indicating docker doesn't know where "rootfs/" subdir is?!

And please don't wonder the complicated setup. I need different versions of php-app during migration-tasks from old to new versions. I also have multiple projects and wanted to recycle the main-app-base (which has contains apache + specific php-mod version) not having to rebuild that everytime from scratch. I have multiple app-version specific docker-compose.yml which are used for extending. So I thought that would be the best way to cover that.

My paths/compose setup looks as follows:

project-root:
docker-compose.yml
  .docker/
    services/
      httpd-php/
        rootfs/     # containing files to be adding using Dockerfile on build
          docker-entrypoint.sh
        Dockerfile.apache2-modphp
        docker-compose.build.yml

the root docker-compose.yml has the build-service using extend like this:

services:
  app-build:
    extends:
      file: ./services/httpd-php//Dockerfile.apache2-modphp
      service: typo3-build
    build:
      args:
        APP_RELEASE=xyz

I also added platforms attribute to docker-compose.build.yml - is this the right way?

services:
  typo3-build:
    image: my-app-base
    # ... other stuff here
    build:
      platforms:
        - linux/amd64

[typoworx-de]

Little update regarding trying to build without docker-compose using "plain" docker with buildx:
docker build --target stage-release -f .docker/services/httpd-php/Dockerfile.apache2-modphp --load .docker/services/httpd-php/

This variant seems to work. My mistake here. But I still have to demystify how to get this working using docker-compose build and which build.context path is used. I'll try now to run the same with docker-compose build (using .docker/services/httpd-php/ as context).

[typoworx-de]

While the build works manually I now discovered my initial assume was correct. The "build.context" seems to be relative to where the docker-compose.yml is located. After changing .docker/services/httpd-php/docker-compose.build.yml with context ".docker/services/httpd-php/" docker-compose build complains just before building:

docker-compose-build
unable to prepare context: path "/media/[my-absolute-project-path]/.docker/services/httpd-php/.docker/services/httpd-php" not found

[typoworx-de]

Ok I think the build for base-image works now. Seems like the culprit is somewhere else .Having the image build "base-image" (named typo3-httpd:v12 in my case) and trying to build Dockerfile.project which should be FROM typo3-httpd:v12

Build command (without docker-compose):

docker build --target release-httpd -f .docker/services/httpd-php/Dockerfile.project --load .docker/services/httpd-php/

This error occurs:

 | >>> FROM typo3-httpd:v12 AS release-httpd

ERROR: failed to solve: typo3-httpd:v12: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

docker image ls exposes that the image exists locally:

docker image ls | grep typo3-httpd
typo3-httpd                                                                                                              v12                     496dfca5fcb4   17 minutes ago   715MB

[typoworx-de]

I think buildkit is the problem. Figured out similar problem has been already mentioned here:

• docker-compose build results in repository pull errors for local images docker/compose#9232
• Cannot give a local image to FROM when using docker-container #2343

buildkit isolation seems to be the problem while having two separate builds whereas one (dockerfile) depends on the other.

[jedevc]

Can you share the results of docker info and docker buildx ls please?

Using local images in builds like this will only work as expected using the docker driver. See #2343.

[typoworx-de]

@jedevc thanks for trying to help. Here is the info.

And just to clarify. The build consists of two services. One generic build-image and a project-based build basing on the first image provided by the other service. On the host the build image of first build-service is loaded. But it's obviously missing as "FROM"-Base for the build inside buildkit.

Actually I'm trying to solve that using a combination of local-registry and cache-from & cache-to. Not shure yet it works (build is actually running). But I think this hazzles should either work out of the box or be documentated better to help developers solving this.

$ docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.11.2)
  compose: Docker Compose (Docker Inc., v2.20.3)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 27
  Running: 21
  Paused: 0
  Stopped: 6
 Images: 112
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: local
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: crun io.containerd.runc.v2 io.containerd.runtime.v1.linux runc sysbox-runc
 Default Runtime: crun
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.15.89-xanmod1
 Operating System: Ubuntu 16.04.7 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 62.6GiB
 Name: gabriel-XMG
 ID: PZ6V:VDD5:IKCG:L27W:Q3BS:VO7L:GVQU:TELK:LD6F:5H3W:TEYI:7UKC
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: typoworx
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.11:5000
  127.0.0.0/8
 Registry Mirrors:
  http://registry-api.php-stack.docker:5000/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 192.168.165.0/24, Size: 24
   Base: 172.30.0.0/16, Size: 24

$ docker buildx ls

NAME/NODE            DRIVER/ENDPOINT             STATUS  BUILDKIT        PLATFORMS
confident_neumann *  docker-container                                    
  confident_neumann0 unix:///var/run/docker.sock running v0.12.1         linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
default              docker                                              
  default            default                     running v0.8.2+244e8cde linux/amd64, linux/386

Actually I'm using this command to setup docker-buildx:

docker buildx create \
    --use --bootstrap \
    --driver docker-container  \
    --driver-opt network=host \
    --driver-opt image=moby/buildkit:latest \
    --config /etc/buildkit/buildkitd.toml

[typoworx-de]

I'm totally lost thinking about dropping buildx and buildkit! This renders useless if even multi-stage build (same Dockerfile!!!) begin to fail like this:

#7 [httpd-base internal] load metadata for docker.io/library/stage-rc:latest
#7 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

[httpd-base internal] load metadata for docker.io/library/stage-rc:latest:

---

failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = stage-rc: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

[typoworx-de]

What a shame ... at least the last problem with missing stage in same dockerfile was solved. What a shame!
it was simply a stupid failure at least for the issue with multi-stage's in same Dockerfile!

I had a Backslash in the last argument of a RUN statement which leads to the interpretation, that the next statement like:

RUN
foo; \ <-- DOH

A comment

FROM ... AS stage2
Would be awesome may be to integrate something like hadolint into build I'm shure it would have found out that there's something wrong ;-)

[typoworx-de]

I'm still trying to get everything working. I'm not shure this is a buildkit or a buildx issue. It's complicated to determine what component is causing the issue without being deep diving into the topic.

I've made a very simple reproducer with alpine image (without real builds inside) and even this is failing like reported:
https://github.com/Gabriel-Kaufmann/reproducer-docker-compose-buildx-issue

And to be shure it's adressed correctly I also made an issue for buildx:
docker/buildx#2024

If there is success on one of the tickets resolving or finding the real culprit I will report in the other ticket.