Can not connect to Ryuk at localhost

[fang10]

I am running into an issue using testcontainer with bitbucket pipeline. The issue may be relevant to recent bitbuket pipeline changes (e.g. #700) since I didn't have this problem a few days ago.

<groupId>org.testcontainers</groupId>
<artifactId>postgresql</artifactId>
<version>1.9.1</version>
<scope>test</scope>

2018-10-02 01:25:30.930  WARN 138 --- [containers-ryuk] o.testcontainers.utility.ResourceReaper  : Can not connect to Ryuk at localhost:32768
java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at java.net.Socket.<init>(Socket.java:434)
	at java.net.Socket.<init>(Socket.java:211)
	at org.testcontainers.utility.ResourceReaper.lambda$start$1(ResourceReaper.java:112)
	at java.lang.Thread.run(Thread.java:745)
2018-10-02 01:26:00.180 ERROR 138 --- [           main] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Exception during pool initialization.

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [org/springframework/boot/autoconfigure/liquibase/LiquibaseAutoConfiguration$LiquibaseConfiguration.class]: Invocation of init method failed; nested exception is com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Can not connect to Ryuk
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1699)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:573)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:759)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:762)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:398)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:330)
	at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:139)
	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99)
	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:117)
	at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:108)
	at org.springframework.test.context.web.ServletTestExecutionListener.setUpRequestContextIfNecessary(ServletTestExecutionListener.java:190)
	at org.springframework.test.context.web.ServletTestExecutionListener.prepareTestInstance(ServletTestExecutionListener.java:132)
	at org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:246)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:227)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:289)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:291)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:246)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
	at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Can not connect to Ryuk
	at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:569)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:555)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
	at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:302)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1758)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1695)
	... 42 common frames omitted
Caused by: java.lang.IllegalStateException: Can not connect to Ryuk
	at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:148)
	at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:65)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:124)
	at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:162)
	at org.testcontainers.containers.JdbcDatabaseContainer.<init>(JdbcDatabaseContainer.java:40)
	at org.testcontainers.containers.PostgreSQLContainer.<init>(PostgreSQLContainer.java:30)
	at org.testcontainers.containers.PostgreSQLContainerProvider.newInstance(PostgreSQLContainerProvider.java:19)
	at org.testcontainers.containers.JdbcDatabaseContainerProvider.newInstance(JdbcDatabaseContainerProvider.java:47)
	at org.testcontainers.jdbc.ContainerDatabaseDriver.connect(ContainerDatabaseDriver.java:94)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:117)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:123)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:365)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:194)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:460)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:534)
	... 47 common frames omitted

[kiview]

Probably related, you are right.
You don't have any other Testcontainers dependency?

[fang10]

Probably related, you are right.
You don't have any other Testcontainers dependency?

Right. so far I am only using a global postgres container(https://www.testcontainers.org/usage/database_containers.html#jdbc-url) in my integration test.

[alex-arana]

I am running into the same issue while building my project in BitBucket pipelines. I can see that the Docker environment is detected during startup with the following message in the logs:

 23:13:09.604 [Test worker] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
        dockerHost=tcp://localhost:2375
        apiVersion='{UNKNOWN_VERSION}'
        registryUrl='https://index.docker.io/v1/'
        registryUsername='root'
        registryPassword='null'
        registryEmail='null'
        dockerConfig='DefaultDockerClientConfig[dockerHost=tcp://localhost:2375,registryUsername=root,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/root/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

However, later on I get the same error as @fang10 (notice the port number is also 32768)

    23:13:10.560 [Test worker] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: quay.io/testcontainers/ryuk:0.2.2
    23:13:10.560 [Test worker] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /root/.docker/config.json (does not exist) and commandPathPrefix: 
    23:13:10.560 [Test worker] WARN org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: quay.io/testcontainers/ryuk:0.2.2, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
    23:13:10.560 [Test worker] DEBUG org.testcontainers.dockerclient.auth.AuthDelegatingDockerClientConfig - Effective auth config [null]
    23:13:10.578 [Test worker] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: com.github.dockerjava.core.command.CreateContainerCmdImpl@be46b6[name=testcontainers-ryuk-1fe5080f-07e1-40dd-a2d1-737d9924192f,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd=<null>,entrypoint=<null>,image=quay.io/testcontainers/ryuk:0.2.2,volumes=com.github.dockerjava.api.model.Volumes@71548ecb,workingDir=<null>,macAddress=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@91ce71c,stopSignal=<null>,hostConfig=com.github.dockerjava.api.model.HostConfig@6f4c4ba5[binds=com.github.dockerjava.api.model.Binds@250c0922,blkioWeight=<null>,blkioWeightDevice=<null>,blkioDeviceReadBps=<null>,blkioDeviceReadIOps=<null>,blkioDeviceWriteBps=<null>,blkioDeviceWriteIOps=<null>,memorySwappiness=<null>,capAdd=<null>,capDrop=<null>,containerIDFile=<null>,cpuPeriod=<null>,cpuShares=<null>,cpuQuota=<null>,cpusetCpus=<null>,cpusetMems=<null>,devices=<null>,diskQuota=<null>,dns=<null>,dnsSearch=<null>,extraHosts=<null>,links=<null>,logConfig=<null>,lxcConf=<null>,memory=<null>,memorySwap=<null>,memoryReservation=<null>,kernelMemory=<null>,networkMode=<null>,oomKillDisable=<null>,autoRemove=true,oomScoreAdj=<null>,portBindings=<null>,privileged=<null>,publishAllPorts=true,readonlyRootfs=<null>,restartPolicy=<null>,ulimits=<null>,volumesFrom=<null>,pidMode=<null>,securityOpts=<null>,cgroupParent=<null>,volumeDriver=<null>,shmSize=<null>,pidsLimit=<null>,runtime=<null>,tmpFs=<null>],labels={org.testcontainers=true},networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=<null>,execution=com.github.dockerjava.core.exec.CreateContainerCmdExec@5f8d15be]
    23:13:10.716 [Test worker] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 788506471ec21fb1c7f34fb9a57998dda0b69c4082e58cdf7368bc0c09a1e562,com.github.dockerjava.core.exec.StartContainerCmdExec@52cb070a
    23:13:10.975 [Test worker] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 788506471ec21fb1c7f34fb9a57998dda0b69c4082e58cdf7368bc0c09a1e562,false,com.github.dockerjava.core.exec.InspectContainerCmdExec@a8a77ce
    23:13:10.975 [Test worker] DEBUG com.github.dockerjava.core.exec.InspectContainerCmdExec - GET: OkHttpWebTarget(okHttpClient=org.testcontainers.shaded.okhttp3.OkHttpClient@f3115df, baseUrl=http://localhost:2375/, path=[/containers/788506471ec21fb1c7f34fb9a57998dda0b69c4082e58cdf7368bc0c09a1e562/json], queryParams={})
    23:13:11.067 [testcontainers-ryuk] WARN org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at localhost:32768
    java.net.ConnectException: Connection refused (Connection refused)
    	at java.net.PlainSocketImpl.socketConnect(Native Method)
    	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    	at java.net.Socket.connect(Socket.java:589)
    	at java.net.Socket.connect(Socket.java:538)
    	at java.net.Socket.<init>(Socket.java:434)
    	at java.net.Socket.<init>(Socket.java:211)
    	at org.testcontainers.utility.ResourceReaper.lambda$start$1(ResourceReaper.java:112)
    	at java.lang.Thread.run(Thread.java:748)

[alex-arana]

I am using latest release (1.9.1) also..

  dependency("org.testcontainers:localstack:1.9.1")

[alex-arana]

I am using latest release (1.9.1) also..

  dependency("org.testcontainers:localstack:1.9.1")

As of this afternoon (AEST), testcontainers:localstack is again not working in BitBucket pipelines. The error is the same as before:

java.lang.ExceptionInInitializerError: java.lang.IllegalStateException: Can not connect to Ryuk

[kiview]

@alex-arana @fang10
Sorry to hear this, but can you maybe raise an issue with Atlassian, as described in this comment?

I'm unsure if this a general incompatibility with Testcontainers, or just something that's rolled out for some Bitbucket users.

[nathanburrell]

This is caused by the following bug within test containers.

#712

The reason why we enabled this feature Is described here:

#700 (comment)

For now you can reach out to Atlassian support so we can add you to an exclusion list whilst test containers fixes this issue.

I'll comment again on the 712, as over the coming months, we will be removing the exclusion list as it leaves potential security holes in our product, and if test containers hasn't fixed this issue by than it will render it unusable.

[jrehwaldt]

Related ticket at Atlassian.

[usommerl]

It seems to me that the ResourceReaper implementation always tries to bind mount the docker unix socket when starting ryuk. This will fail in environments where only the tcp socket is available.

testcontainers-java/core/src/main/java/org/testcontainers/utility/ResourceReaper.java

Lines 72 to 84 in 58a6c69

	List<Bind> binds = new ArrayList<>();
	binds.add(new Bind("//var/run/docker.sock", new Volume("/var/run/docker.sock")));
	String ryukContainerId = client.createContainerCmd(ryukImage)
	.withHostConfig(new HostConfig().withAutoRemove(true))
	.withExposedPorts(new ExposedPort(8080))
	.withPublishAllPorts(true)
	.withName("testcontainers-ryuk-" + DockerClientFactory.SESSION_ID)
	.withLabels(Collections.singletonMap(DockerClientFactory.TESTCONTAINERS_LABEL, "true"))
	.withBinds(binds)
	.withPrivileged(TestcontainersConfiguration.getInstance().isRyukPrivileged())
	.exec()
	.getId();

[fdeysel]

I am running into the same issue while building my project in BitBucket pipelines. Are there any updates regarding this issue?

[3t-dev]

I faced with this issue at version 1.8.3 although it worked in the past and no firewall constrain.
But it work well when I change version up to 1.9.1.

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>testcontainers</artifactId>
    <version>1.9.1</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>mssqlserver</artifactId>
    <version>1.9.1</version>
    <scope>test</scope>
</dependency>

Hope it help.

[rnorth]

We're aware of this issue, however, I'll have to close this as a duplicate of #700. Please follow that ticket for more updates.