Docker compose pull doesn't respect local images

[antony]

During my searches I've seen this issue raised and closed a few times, but even using the latest build of docker-compose, this still doesn't behave the way I understand that it is expected to.

Scenario:
We're building a composed environment for our CI tests to run. We grab a database image, an api image, and a local copy of the image containing application we are trying to test (it has been built in a previous CI step, and tagged with the build hash provided by our CI environment (CIRCLE_SHA1)

docker-compose.yml

api:
  image: mycompany/api

undertest:
  image: mycompany/undertest:${CIRCLE_SHA1}

db:
  image: mycompany/db

The commands we run then are as follows:

docker-compose pull # Because we definitely want the latest db, and api for a true test.
docker-compose up # compose the environment

Actual Result:
No matter what I do, docker compose always tries to pull my CIRCLE_SHA1 tagged version from docker hub. it doesn't exist there, I never want to push it (until it passes tests and is re-tagged as :latest and/or :release

I have a unique tag CIRCLE_SHA1 which only exists inside the build environment, meaning no confusion for docker-compose when it tries to pull, and yet, it seems to try to fetch it anyway, and fail even though that exact tag exists locally.

Expected Result:
I'd expect the fact that there is no remote build tagged with CIRCLE_SHA1 to cause docker-compose to use the local copy it finds. I need to do pull, because I want everything else to be the latest.

I'd suggest that if there is confusion where image: refers to a remote repository, then perhaps we could use local: instead, to reference a local image?

[decoomanj]

I'm facing similar issues. I use the --ignore-pull-failures option when I do a docker-compose pull. This basically ignores ALL errors. It is ok as a workaround. We do the same thing in our pipeline. I pull other production-images and start my freshly build release-candidate container locally. When the tests are ok, it's uploaded.

[johnharris85]

I don't see this is a issue / bug with current functionality. pull is doing exactly that, how would it know in your original compose file that 2 images should be pulled remotely but one should be grabbed locally (seeing as they all probably exist locally right, you just want to 'refresh' the other two?).

Perhaps a --filter option to pull? So something like:

docker-compose.yml
version: '2'
services:
  api:
    image: mycompany/api
    labels:
      - "refresh"
  undertest:
    image: mycompany/undertest:${CIRCLE_SHA1}
  db:
    image: mycompany/db
    labels:
      - "refresh"

Then:

docker-compose pull --filter "label=refresh"

@aanand @dnephin thoughts?

[dnephin]

--ignore-pull-failures was added for this very reason.

You could also do docker-compose pull db api to only pull the ones you wanted.

[johnharris85]

Both of those options seem kind of sub-optimal.

It's not really a 'failure' to ignore, more of a separate use case.

The second option means you're having to declare some behavior / state outside of the compose file. I.e. I have all of this automated but then want to change which images are pulled and which aren't, I probably want that to be reflected in my compose file as a single-point-of-truth, but I have to edit the pull command instead.

[antony]

I'd agree with @johnharris85 on --ignore-pull-failures not feeling like the option to use. Something like --prefer-local would work - but I'd still probably prefer a concrete local: my-org/my-image:tag to make for a clearer intent.

I didn't know you could pull specific images, so thanks for that @dnephin - the downside being, this would move knowledge about the composure of my stack outside the compose file, whereas for me a compose file is a one-stop encapsulation of my architecture.

[dnephin]

The second option means you're having to declare some behavior outside of the compose file.

Yes, and in this case I think that's correct. The behaviour that is external to the Compose file is not the behaviour of a container, but part of your project dev workflow. Compose is not a project workflow tool. It's a tool for orchestrating containers.

The Compose file should really only include configuration for those containers. How to pull an image is not container configuration, it's part of the workflow, and currently all of those options are commands and flags. I think this is an important distinction. Trying to mix both would make the Compose file very confusing.

The equivalent of --prefer-local is to just never run docker pull in the first place. Any missing images will be pulled before trying to run the container.

[JashaadGaines]

Problem: docker-compose -f docker-compose.yml up -> doesn't look locally for the image at all.
I agree @johnharris85 - These options are suboptimal. Docker is sort of acting like a dependency manager or at least one dependency resolver. It pulls images from a specified/default repo then/and only then, commences it's docker process. In the same way gradle, ant, maven, gulp, npm, etc. pull dependencies and define processes the user can run. The standard has been set for Dependency Mgr/Resolvers, look locally first (sometimes cache or something), then pull from repo. I'm trying to think of a really pressing reason why docker would want to force you to look at the repo first. Just like @antony & @decoomanj I don't want to push any images that don't even satisfy our lowest level of quality threshold: unit tests. Not to mention any other test suite.

Suggestion:
rancher-compose offers a --pull option to insure you're getting the latest image from the repo. It would be great if docker-compose looked for images locally first then to the repo. And if necessary there can be a --pull option to force docker to only search for images in the repo.

[martindariocernadas]

Any clues how to avoid this issue?

[smebberson]

I'd like to know how to avoid this issue too.

[robinjhector]

I'd also like to use a locally built image, before building and publishing a real image to our registry. How would I go about doing that? Looking at this conversation, it seems impossible to use locally installed images?

[KamilKopaczyk]

@dnephin

The Compose file should really only include configuration for those containers. How to pull an image is not container configuration

It's not only for containers if docker-compose allows to pull and push.

Maybe it shouldn't be concern for docker-compose? Maybe docker should mark locally built, not-yet-pushed images and ignore it when trying to pull this particular image.

It's PITA since latest docker-compose releases, we're able to build docker images with concrete name specified in docker-compose.yml.
This way if you do docker-compose build --pull you end up with errors and images not being pulled and built. It's even worse - you can't use --ignore-pull-failures flag with docker-compose build.

[skehlet]

A hacky workaround to use your local images is to run your own registry:

docker run -d -p 5000:5000 --name registry registry:2
docker tag 4e6a104bad19 localhost:5000/stevek/my-container
docker push localhost:5000/stevek/my-container
# edit docker-compose file to use localhost:5000/stevek/my-container
docker-compose ...

[Constantin07]

+1 same issue

[karunapuri]

Try this './docker-env -c ./build --no-update'. This command works for running locally built images.

[joeyhub]

This kind of solution isn't really viable:

 --ignore-pull-failures was added for this very reason.

People want to ignore that an image is local. Not necessarily every other type of error.

Developers aiming to create robust systems will program strictly and abort on the first error. This breaks with that.

In nearly all my years of programming auto-fix, force and ignore options turn out useless if you program safely. They tend to only be genuinely useful in 0.0001% of cases for safe code.

If it's to be useful, then it should only fire when the error is non-critical (as in it has the image, only update failed). That's or there should be a variant that does that.

It should be possible to have something such as pull --missing-only. That would at least allow a pre-fetch.

It doesn't seem like docker stores enough information on images to know if they are local or remote.

The correct way to support it would be to add an option in the docker compose file that an image is built locally.

There's the question of why this happens at all if docker-compose can build images (and then knows that they are local).

For me it's because it isn't very good at that (for example it doesn't have a separate build section, instead it's crammed into services), I've made my own tool to manage and wrap that.

This then breaks pull because docker-compose doesn't know about them being built locally.

I can implement the functionality I need with my system. At that point docker-compose wont be used for build or pull at all for my, only service management (run, stop, up, down, start).

It would be nice though if docker compose fixed these issues at the core. It is also missing an image section where you should be able to specify images that are local and some management for private repositories.

If you have a repository section then you could specify images for those repositories (is left out maybe have it automatic, then some thought going into what happens when an image is in the global and one or more private).

repositories:
     local:
           url: ~ (maybe blank for local)
           images: [a, b, c] (simplest)
           namespace: company (works dependent on how you're tagging your images, matches company/db, can be more complex like a/b/c)

[noaho]

With the status quo, using —ignore-pull-failures and —include-deps together doesn’t actually work.. my dockerfile has “FROM: ubuntu:16.04”, but this dependency is never pulled because of the pull failure...

Of course it is pulled when I later build, but there is no way to force pull the latest version of every dependency, so it is left with the first version of the image forever...

I imagine there is a lot of out of date code in production for this reason..

[zuernBernhard]

--prefer-local (prefer newer image with same tag on local machine) would be great - than we can use the same compose file and workflow in local development and in production - and it is always possible to locally override images with ease for testing purpose

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[SuperSandro2000]

Bump

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[iamarkadyt]

Bump

[amackiewicz]

bump

[SuperSandro2000]

I think this syntax is more logical.

services:
  my_service:
    image: some_external image
  my_private_service:
    image:
      name: some_local_only_image
      pull: false

[mslipets]

Bump!

[akashjain971]

A workaround for this while developing locally to save pull time:

1. Create a local docker registry like this
2. Pull the image from the remote docker registry (abc/xyz:latest)
3. Tag this image to point to the local docker registry (docker tag abc/xyz:latest localhost:5000/abc-xyz-latest)
4. Push this image to the local docker registry (docker push localhost:5000/abc-xyz-latest)
5. Use this image in your local docker-compose file

[EricBorland]

@akashjain971 Oh man, you saved me. It's a very ugly workaround but works like a charm. Shame than docker-compose doesn't have anything like the pull: false commented above.
Thank you very much :)

[thespad]

Is it not simpler for there to be a reserved "repo" name for locally-built images such that something like:

service:
  my_service:
    image: localhost/myimage:devtag
  my_other_service:
    image: redis:alpine

Means that Compose knows that my_service is using an image that only exists as a local build and would never attempt to pull it?

[tdepalle]

Still no solution better than creating a local registry ?

[erdnaxeli]

@ard-i that's not what the issue is about. The issue is about using docker-compose pull when at least one image in the docker-compose file does not exist on the remote registry but exists locally.

[haneef95]

Thanks @lucasbasquerotto

Using build: worked like a charm!

[ndeloof]

AFAICT pull_policy: missing introduced by the compose specification should be enough to address this issue.

[lucasbasquerotto]

@ndeloof With Compose implementations SHOULD pull the image only if it's not available in the platform cache it means that if an image with the tag is already local it won't even try to pull (nor try to connect to the remote repository), or it will try to verify if the remote image with the tag exists and has the same image id/hash? (because a new image can be pushed to a remote registry with the same tag as an older image, replacing it)

The former is probably what people in this issue want (no remote connection at all if image:tag exists locally). The later is what already happens currently.

[ndeloof]

The spec don't define how implementation has to behave regarding registry connexion, but a reasonable implementation would be:

exists := localengine.lookup(image)
if ! exists {
    docker.pull(image)
}

About checking the image is up-to-date with registry one, there's unfortunately no such thing as a portable hash to be compared with the remote. The local image ID only exists in the local engine. Using always would then force (at least, try to) pull from registry:

err := docker.pull(image)
if err != nil {
	exists := localengine.lookup(image)
	if ! exists {
		return error.New("Sorry")
    }	
    // never mind, we have a local image for it. Maybe just warn user?
}

[haneef95]

AFAICT pull_policy: missing introduced by the compose specification should be enough to address this issue.

Thanks @ndeloof

I've tried both pull_policy: and pull: defined in docker run.

Both are giving me errors:

docker-compose version 1.25.0

[ndeloof]

I said this was introduced by the compose-spec, and could be used to support this feature request, not that it's already supported in docker compose!

[haneef95]

hopefully, it's implemented soon! This is already available with docker run, so should be a fairly quick implementation?

[ndeloof]

docker run is a complete distinct beast actually. But yes indeed, could be implemented quickly as the structure is already defined in the compose-spec model

[eladitzhakian]

Is this being addressed? It's a real pain for some use cases and the solution seems pretty straightforward

[gsmari]

for me I needed to specify the local IMAGE ID instead of the name of the image in docker-compose.yml

Instead of

image: "export-job"

I changed it to

image: "80bec021cb5e"

and then running sudo docker compose up -d --no-deps --build

EDIT:
This is not needed if local images are tagged accordingly, matching the image definition in docker-compose.yml

[filinvadim]

My workaround:
DOCKER_BUILDKIT=0 docker compose -f docker-compose.yml up your-app

[filinvadim]

Found the issue. For latest Docker versions that happens because you haven't pointed base image platform type like linux/amd64 for example.

[erlangparasu]

@filinvadim thanks. add DOCKER_BUILDKIT=0 to environment works for me

[typoworx-de]

Still having the issue and found this thread! I don't think disabling Buildkit is a smart hack if one wants to use buildkit-features like COPY --chmod=a+x

The matter that buildkit obviously breaks this is a bug. Has this been reported to buildkit or is the bug located in docker itself?

[al-dot-exe]

Know this issue is closed but you can use watchtower on a custom custom interval to keep remote images up to date and never push your local image to a remote registry.