Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker-compose build <service-name> not following depends_on order #6332

Closed
ArtyMcFlyHA opened this issue Nov 8, 2018 · 11 comments
Closed

Comments

@ArtyMcFlyHA
Copy link

ArtyMcFlyHA commented Nov 8, 2018

Description of the issue

docker-compose build <service-name> does not appear to follow depends_on like docker-compose up -d <service-name>. Note, this is different than #5228 which has a similar wording, but the linked issue does not use depends_on in the docker-compose.yml file.

Context information (for bug reports)

The Dockerfiles in question:

FROM alpine:3.8
RUN apk update && apk upgrade
WORKDIR /home
FROM my-private-registry/alpine:3.8
COPY npmrc /home/.npmrc
RUN apk add nodejs-current && node -v
WORKDIR /home

Output of docker-compose version

docker-compose version 1.22.0, build f46880f

Output of docker version

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:21:31 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:29:02 2018
  OS/Arch:          linux/amd64
  Experimental:     true

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

services:
  alpine:
    build:
      context: /Users/aweborg/Documents/workspace/Utilities/dockerImages/alpine
    image: my-private-registry/alpine:3.8
  nodejs:
    build:
      context: /Users/aweborg/Documents/workspace/Utilities/dockerImages/nodejs
    depends_on:
    - alpine
    image: my-private-registry/nodejs:9.11
version: '3.0'

Steps to reproduce the issue

  1. (empty registry for both images) run docker system prune -a (to ensure no local images cached)
  2. run docker-compose build nodejs

Observed result

nodejs is attempted to be built without attempting to build alpine first and an error occurs as a result. (stacktrace below)

Expected result

I expected alpine to build first, nodejs to build second. If docker-compose up -d nodejs is run instead of docker-compose build nodejs than alpine is built first as expected. I am expecting the following output:

Building alpine
Step 1/3 : FROM alpine:3.8
3.8: Pulling from library/alpine
4fe2ade4980c: Pull complete
Digest: sha256:621c2f39f8133acb8e64023a94dbdf0d5ca81896102b9e57c0dc184cadaf5528
Status: Downloaded newer image for alpine:3.8
 ---> 196d12cf6ab1
Step 2/3 : RUN apk update && apk upgrade
 ---> Running in 8cc45ac07394
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
v3.8.1-59-ge104e0f441 [http://dl-cdn.alpinelinux.org/alpine/v3.8/main]
v3.8.1-42-ge6bc061baf [http://dl-cdn.alpinelinux.org/alpine/v3.8/community]
OK: 9546 distinct packages available
OK: 4 MiB in 13 packages
Removing intermediate container 8cc45ac07394
 ---> 9fb308b80a88
Step 3/3 : WORKDIR /home
 ---> Running in 55e07020306e
Removing intermediate container 55e07020306e
 ---> 1d0b0417c6de
Successfully built 1d0b0417c6de
Successfully tagged my-private-registry/alpine:3.8
Building nodejs
Step 1/4 : FROM my-private-registry/alpine:3.8
 ---> 1d0b0417c6de
Step 2/4 : COPY npmrc /home/.npmrc
 ---> 7ea90e6de2a0
Step 3/4 : RUN apk add nodejs-current && node -v
 ---> Running in 740846cfb3fa
(1/9) Installing ca-certificates (20171114-r3)
(2/9) Installing c-ares (1.14.0-r0)
(3/9) Installing libcrypto1.0 (1.0.2p-r0)
(4/9) Installing libgcc (6.4.0-r9)
(5/9) Installing http-parser (2.8.1-r0)
(6/9) Installing libssl1.0 (1.0.2p-r0)
(7/9) Installing libstdc++ (6.4.0-r9)
(8/9) Installing libuv (1.20.2-r0)
(9/9) Installing nodejs-current (9.11.1-r2)
Executing busybox-1.28.4-r1.trigger
Executing ca-certificates-20171114-r3.trigger
OK: 31 MiB in 22 packages
v9.11.1
Removing intermediate container 740846cfb3fa
 ---> ab51418bf940
Step 4/4 : WORKDIR /home
 ---> Running in ee83f6b41566
Removing intermediate container ee83f6b41566
 ---> d1d7a9a120ba
Successfully built d1d7a9a120ba
Successfully tagged my-private-registry/nodejs:9.11

Stacktrace / full error message

> docker-compose build nodejs
Building nodejs
Step 1/4 : FROM my-private-registry/alpine:3.8
ERROR: Service 'nodejs' failed to build: manifest for my-private-registry/alpine:3.8 not found

Additional information

macOS High Sierra version 10.13.6

Note, docker-compose build appears to work as expected (though that may be due to alphabetical ordering of service names?). However, the above is a boiled down least code replication of the issue. A more practical example of where this issue becomes a problem is when there are several dozen services and only 1 or 2 need to explicitly have their images/dependencies built.

Also note, I replaced the private registry domain I'm using with my-private-registry. If you use a made up repository, the error will read Service 'nodejs' failed to build: pull access denied for my-private-registry/alpine, repository does not exist or may require 'docker login' instead of ... failed to build: ... not found. That said, expected output should be the same as above.

@ArtyMcFlyHA ArtyMcFlyHA changed the title docker-compose build not following depends_on order docker-compose build <service-name> not following depends_on order Nov 8, 2018
@shin-
Copy link

shin- commented Nov 8, 2018

Hi @ArtyMcFlyHA

The effects of the depends_on key are clearly outlined in the documentation. As you can see, build ordering is not an intended usage of depends_on. More generally, higher-level build automation is not an intended usage of Compose, and we recommend finding complementary tools to fulfill this usage (dobi is usually a good candidate)

@ArtyMcFlyHA
Copy link
Author

ArtyMcFlyHA commented Nov 8, 2018

Hi @shin-

Thanks for the quick response as well as the clarity you've provided! I will definitely look into dobi, thank you for the recommendation! :)

I will say though, I believe as written at the time of this issue creation, we see the documentation painting different pictures. IMHO, it does not clearly outline that build ordering is not an intended usage. https://docs.docker.com/compose/compose-file/#depends_on the "simple example" even includes a service that has a build context.

I realize documentation is a separate issue, but I'd advise that "simple example" have it's build context removed and There are several things to be aware of when using depends_on have the case of docker-compose build be added to it.

I'm not alone in thinking this should've worked, I tracked down a couple posts on other internet forums on the topic where other devs also believe that depends_on would dictate build order. I'm going to link your response to said third party forums.

@codenjoyme
Copy link

This solution can help

@tswaters
Copy link

tswaters commented Aug 23, 2019

There is nothing in the docs about what the effects of depends_on are in the context of docker build. The way it's written, docker build is not even mentioned - to me it's not immediately clear if this is an omission, or a lack of support.

Further, docker build does respect depends_on while docker build <service> does not. Maybe it's a "feature" of docker build, or maybe docker build <service> has a bug. Nothing is mentioned one way or the other.

@lhuhn-wish
Copy link

The effects of the depends_on key are clearly outlined in the documentation. As you can see, build ordering is not an intended usage of depends_on. More generally, higher-level build automation is not an intended usage of Compose, and we recommend finding complementary tools to fulfill this usage (dobi is usually a good candidate)

@shin- Is dobi still recommended? I see that dobi has not received commits since July. I am trying to evaluate what tool to use because we have outgrown docker-compose build.

@rayzorben
Copy link

I am here because like others I feel the documentation is lacking. I spent a couple hours because of this simple example on https://docs.docker.com/compose/compose-file/

version: "3.7"
services:
  web:
    build: .
    depends_on:
      - db
      - redis
  redis:
    image: redis
  db:
    image: postgres

There are no comments that lead me to believe that the behavior is anything but

  1. create network
  2. start db
  3. start redis
  4. build web - since db and redis should be running, the network should be available

@jgonzalezd
Copy link

Wow! So build won't build the services in which it depends_on like up. That's a bit of a suprise. Good to know. I'm glad I found this thread.

@stale
Copy link

stale bot commented Nov 5, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Nov 5, 2020
@stale
Copy link

stale bot commented Nov 12, 2020

This issue has been automatically closed because it had not recent activity during the stale period.

@stale stale bot closed this as completed Nov 12, 2020
@Neirda24
Copy link

Hi, I'd like to re-open this issue as I think having a way to prioritize docker builds would be very helpful.
Sometimes we have a docker-compose file where we build several images. One "core" (nginx for example) and others that will extend it by adding some specific configurations (php for example). When doing so, one has to be built before others. For quite some time (several years) docker-compose build was respecting that order. Suddenly when switching to parallel builds it does not anymore. I understand it wasn't the intend at first but has been widely used... Would it be possible to consider it again ?

Thanks.

@strophy
Copy link

strophy commented May 11, 2022

I think it is now possible to prioritize builds correctly using the buildx bake command, as of buildx 0.8.x. See the "Create Build Pipelines by Linking bake Targets" section in this blog post: https://www.docker.com/blog/dockerfiles-now-support-multiple-build-contexts/

This does not seem to be supported in compose syntax, it is necessary to use the json/hcl bake syntax instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants