"No such image: alpine:3.5" error on health check in Fedora 29 with Docker/Moby 18.06.0-dev

[szpak]

Fedora 29 provides the quite old patched version of Docker 1.13 and Docker (Moby) 18.06.0-dev (1.38) packaged as moby-engine. With Docker 1.13 it is possible to use different containers using different access methods (JUnit 4 rule, Spock, or new GenericContainer()). However, there are some problems with Ryuk (#1356) and 1.13 is quite old. After switching to Docker/Moby 18.06.0-dev I'm no longer able to do anything with Testcontainers.

Even simple:

public class IssueWithMobyOnFedoraReproducer {

    public static void main(String[] args) throws InterruptedException {
        try (GenericContainer redis = new GenericContainer("redis:3-alpine")
                            .withExposedPorts(6379)) {
            redis.start();
            System.out.println("Started");
            sleep(5000);
        }
    }
}

finishes with:

com.github.dockerjava.api.exception.NotFoundException:
    {"message":"No such image: alpine:3.5"}

Docker in general seems to work fine:

$ docker version                                                                                22:32
Client:
 Version:           18.06.0-dev
 API version:       1.38
 Go version:        go1.11.5
 Git commit:        0ffa825
 Built:             Mon Feb 11 14:47:59 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          dev
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.11.5
  Git commit:       0ffa825
  Built:            Mon Feb 11 14:47:26 2019
  OS/Arch:          linux/amd64
  Experimental:     false

$ docker images | grep alpine
postgres                              10-alpine           b97fc2e43226        2 weeks ago         70.8MB
postgres                              11-alpine           fd0b93397ff8        2 weeks ago         70.8MB
alpine                                3.5                 f80194ae2e0c        4 months ago        4MB
redis                                 3-alpine            6e94a98d3442        8 months ago        22.9MB

I can use/run images from the command line.

The full logs with a stacktrace (pointing to some health check operations) with testcontainers and the docker client set to debug:

2019-06-18 22:38:00.627 DEBUG   --- [           main] o.t.utility.TestcontainersConfiguration  : Testcontainers configuration overrides will be loaded from file:/home/foobar/.testcontainers.properties
2019-06-18 22:38:00.633 DEBUG   --- [           main] o.t.utility.TestcontainersConfiguration  : Testcontainers configuration overrides loaded from TestcontainersConfiguration(properties={docker.client.strategy=org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy})
2019-06-18 22:38:00.669  INFO   --- [           main] o.t.d.DockerClientProviderStrategy       : Loaded org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy from ~/.testcontainers.properties, will try it first
2019-06-18 22:38:00.724  INFO   --- [           main] o.t.d.DockerClientProviderStrategy       : Will use 'okhttp' transport
2019-06-18 22:38:01.147 DEBUG   --- [     ducttape-0] o.t.d.DockerClientProviderStrategy       : Pinging docker daemon...
2019-06-18 22:38:01.464 DEBUG   --- [     ducttape-0] c.g.d.core.command.AbstrDockerCmd        : Cmd: org.testcontainers.dockerclient.transport.okhttp.OkHttpDockerCmdExecFactory$1@7f6a577d
2019-06-18 22:38:01.634  INFO   --- [           main] tAndSystemPropertyClientProviderStrategy : Found docker client settings from environment
2019-06-18 22:38:01.650  INFO   --- [           main] o.t.d.DockerClientProviderStrategy       : Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=unix:///var/run/docker.sock
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='foobar'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=foobar,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/home/foobar/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

2019-06-18 22:38:01.653  INFO   --- [           main] org.testcontainers.DockerClientFactory   : Docker host IP address is localhost
2019-06-18 22:38:01.656 DEBUG   --- [           main] c.g.d.core.command.AbstrDockerCmd        : Cmd: com.github.dockerjava.core.exec.InfoCmdExec@156b88f5
2019-06-18 22:38:01.868 DEBUG   --- [           main] c.g.d.core.command.AbstrDockerCmd        : Cmd: com.github.dockerjava.core.exec.VersionCmdExec@338fc1d8
2019-06-18 22:38:01.877  INFO   --- [           main] org.testcontainers.DockerClientFactory   : Connected to docker: 
  Server Version: dev
  API Version: 1.38
  Operating System: Fedora 29 (Workstation Edition)
  Total Memory: 15946 MB
        ℹ︎ Checking the system...
        ✔ Docker version should be at least 1.6.0
2019-06-18 22:38:01.887 DEBUG   --- [           main] c.g.d.core.command.AbstrDockerCmd        : Cmd: ListImagesCmdImpl[imageNameFilter=alpine:3.5,showAll=false,filters=com.github.dockerjava.core.util.FiltersBuilder@0,execution=com.github.dockerjava.core.exec.ListImagesCmdExec@821330f]
2019-06-18 22:38:01.920 DEBUG   --- [           main] o.t.utility.RegistryAuthLocator          : Looking up auth config for image: alpine:3.5
2019-06-18 22:38:01.921 DEBUG   --- [           main] o.t.utility.RegistryAuthLocator          : RegistryAuthLocator has configFile: /home/foobar/.docker/config.json (does not exist) and commandPathPrefix: 
2019-06-18 22:38:01.921  WARN   --- [           main] o.t.utility.RegistryAuthLocator          : Failure when attempting to lookup auth config (dockerImageName: alpine:3.5, configFile: /home/foobar/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/foobar/.docker/config.json (No such file or directory)
2019-06-18 22:38:01.922 DEBUG   --- [           main] o.t.d.a.AuthDelegatingDockerClientConfig : Effective auth config [null]
2019-06-18 22:38:01.955 DEBUG   --- [           main] c.g.d.core.command.AbstrDockerCmd        : Cmd: com.github.dockerjava.core.command.CreateContainerCmdImpl@791d1f8b[name=testcontainers-checks-a58c9f2d-2b34-467a-a4c5-9a714a2ade20,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd={tail,-f,/dev/null},entrypoint=<null>,image=alpine:3.5,volumes=com.github.dockerjava.api.model.Volumes@13bc8645,workingDir=<null>,macAddress=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@24c22fe,stopSignal=<null>,hostConfig=com.github.dockerjava.api.model.HostConfig@93081b6[binds=<null>,blkioWeight=<null>,blkioWeightDevice=<null>,blkioDeviceReadBps=<null>,blkioDeviceReadIOps=<null>,blkioDeviceWriteBps=<null>,blkioDeviceWriteIOps=<null>,memorySwappiness=<null>,capAdd=<null>,capDrop=<null>,containerIDFile=<null>,cpuPeriod=<null>,cpuShares=<null>,cpuQuota=<null>,cpusetCpus=<null>,cpusetMems=<null>,devices=<null>,diskQuota=<null>,dns=<null>,dnsSearch=<null>,extraHosts=<null>,links=<null>,logConfig=<null>,lxcConf=<null>,memory=<null>,memorySwap=<null>,memoryReservation=<null>,kernelMemory=<null>,networkMode=<null>,oomKillDisable=<null>,autoRemove=true,oomScoreAdj=<null>,portBindings=<null>,privileged=<null>,publishAllPorts=<null>,readonlyRootfs=<null>,restartPolicy=<null>,ulimits=<null>,volumesFrom=<null>,pidMode=<null>,securityOpts=<null>,cgroupParent=<null>,volumeDriver=<null>,shmSize=<null>,pidsLimit=<null>,runtime=<null>,tmpFs=<null>],labels={org.testcontainers=true, org.testcontainers.sessionId=a58c9f2d-2b34-467a-a4c5-9a714a2ade20},networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=<null>,execution=com.github.dockerjava.core.exec.CreateContainerCmdExec@48b67364]
Exception in thread "main" com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: alpine:3.5"}

	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:270)
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:254)
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.post(OkHttpInvocationBuilder.java:125)
	at com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:33)
	at com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13)
	at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
	at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
	at com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:189)
	at org.testcontainers.DockerClientFactory.runInsideDocker(DockerClientFactory.java:245)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:136)
	at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:138)
	at com.example.demo.Reproduce.main(Reproduce.java:10)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.intellij.rt.execution.application.AppMainV2.main(AppMainV2.java:131)

Process finished with exit code 1

[kiview]

Hey Marcin,

so you didn't install Docker using the Docker repository, but instead installed moby (as moby-engine package) from the Fedora repository, is this correct?

I assume the problem happens when using this image:

testcontainers-java/core/src/main/java/org/testcontainers/utility/TestcontainersConfiguration.java

Lines 45 to 47 in 40724a0

	public String getTinyImage() {
	return (String) properties.getOrDefault("tinyimage.container.image", "alpine:3.5");
	}

Just to be sure (and since I don't know which side effects this switch of Docker engine might have had), could you try to remove your old images and let Docker (by running your Testcontainers code) pull them again? Maybe something with the caching and lookup keys is messed up.

Also, do you still have Ryuk disabled in your setup? I think your problem happens here, which is only executed if Ryuk is not running:

testcontainers-java/core/src/main/java/org/testcontainers/DockerClientFactory.java

Line 136 in ca4505a

runInsideDocker(

[szpak]

Thanks Kevin for you suggestion and sorry for delay. I checked it with Ryuk enabled and the health check passes, but the requested container is not found:

2019-07-11 22:33:54.581  INFO   --- [           main] o.t.d.DockerClientProviderStrategy       : Loaded org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy from ~/.testcontainers.properties, will try it first
2019-07-11 22:33:54.637  INFO   --- [           main] o.t.d.DockerClientProviderStrategy       : Will use 'okhttp' transport
2019-07-11 22:33:55.462  INFO   --- [           main] tAndSystemPropertyClientProviderStrategy : Found docker client settings from environment
2019-07-11 22:33:55.485  INFO   --- [           main] o.t.d.DockerClientProviderStrategy       : Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=unix:///var/run/docker.sock
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='foobar'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=foobar,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/home/foobar.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

2019-07-11 22:33:55.487  INFO   --- [           main] org.testcontainers.DockerClientFactory   : Docker host IP address is localhost
2019-07-11 22:33:55.709  INFO   --- [           main] org.testcontainers.DockerClientFactory   : Connected to docker: 
  Server Version: dev
  API Version: 1.38
  Operating System: Fedora 29 (Workstation Edition)
  Total Memory: 15946 MB
2019-07-11 22:33:56.580  INFO   --- [           main] org.testcontainers.DockerClientFactory   : Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
        ℹ︎ Checking the system...
        ✔ Docker version should be at least 1.6.0
        ✔ Docker environment should have more than 2GB free disk space
2019-07-11 22:33:56.904  INFO   --- [           main] 🐳 [redis:3-alpine]                      : Creating container for image: redis:3-alpine
2019-07-11 22:33:56.922 ERROR   --- [           main] 🐳 [redis:3-alpine]                      : Could not start container

java.lang.reflect.UndeclaredThrowableException: null
	at com.sun.proxy.$Proxy9.exec(Unknown Source)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:230)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:212)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:76)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:210)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:199)
	at com.example.demo.Repro1.main(Repro1.java:12)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.intellij.rt.execution.application.AppMainV2.main(AppMainV2.java:131)
Caused by: java.lang.reflect.InvocationTargetException: null
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testcontainers.dockerclient.AuditLoggingDockerClient.lambda$wrappedCommand$14(AuditLoggingDockerClient.java:98)
	... 12 common frames omitted
Caused by: com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: redis:3-alpine"}

	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:270)
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:254)
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.post(OkHttpInvocationBuilder.java:125)
	at com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:33)
	at com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13)
	at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
	at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
	at com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:189)
	... 17 common frames omitted

Exception in thread "main" org.testcontainers.containers.ContainerLaunchException: Container startup failed
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:217)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:199)
	at com.example.demo.Repro1.main(Repro1.java:12)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.intellij.rt.execution.application.AppMainV2.main(AppMainV2.java:131)
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:83)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:210)
	... 7 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:277)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:212)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:76)
	... 8 more
Caused by: java.lang.reflect.UndeclaredThrowableException
	at com.sun.proxy.$Proxy9.exec(Unknown Source)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:230)
	... 10 more
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testcontainers.dockerclient.AuditLoggingDockerClient.lambda$wrappedCommand$14(AuditLoggingDockerClient.java:98)
	... 12 more
Caused by: com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: redis:3-alpine"}

	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:270)
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:254)
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.post(OkHttpInvocationBuilder.java:125)
	at com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:33)
	at com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13)
	at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
	at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
	at com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:189)
	... 17 more

Process finished with exit code 1

The container seemed to be available:

$ docker images | grep redis
redis                                 3-alpine            6e94a98d3442        9 months ago        22.9MB
redis                                 3.2.12-alpine3.8    6e94a98d3442        9 months ago        22.9MB

However,

$ docker run --rm -it redis:3-alpine                                                                                                                                                                                             22:39
Unable to find image 'redis:3-alpine' locally       <------------
3-alpine: Pulling from library/redis
Digest: sha256:e9083e10f5f81d350a3f687d582aefd06e114890b03e7f08a447fa1a1f66d967
Status: Downloaded newer image for redis:3-alpine
...
1:M 11 Jul 20:39:30.822 * The server is now ready to accept connections on port 6379

and after that I'm able to run Redis with that container with Testcontainers and Moby. The images not available locally works out-of-box.

With Ryuk disable the situation was the same. I needed to try to run the container from a command line to force its download. After that Testcontainers works fine. It would be nice to have Testcontainers download it for me, but it's a rare case, so I'm closing this issue.

[bsideup]

@szpak
Interesting. Testcontainers does download the images.

Server Version: dev

It seems that you're running a dev version of Docker, could you please try the latest stable?

[caicarabruno]

HI all, i was experienced the same issue, i upgraded the test container to the version 1.17.3 and work.