Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Codeship builds fail: can not connect to Ryuk #581

Open
gavinmh opened this issue Feb 10, 2018 · 25 comments
Open

Codeship builds fail: can not connect to Ryuk #581

gavinmh opened this issue Feb 10, 2018 · 25 comments

Comments

@gavinmh
Copy link

gavinmh commented Feb 10, 2018

TestContainers works locally. Builds on Codeship fail with the following:

2018-02-10_19:22:35.027 WARN  o.t.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:32768
java.net.SocketException: Broken pipe (Write failed)
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:134)
	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:136)
	at java.lang.Thread.run(Thread.java:748)
2018-02-10_19:22:35.044 WARN  o.t.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:32768
java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:210)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.readLine(BufferedReader.java:324)
	at java.io.BufferedReader.readLine(BufferedReader.java:389)
	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:139)
	at java.lang.Thread.run(Thread.java:748)

I do not use volume mapping for this service, but I use volumes to build other services. My understanding of https://www.testcontainers.org/usage/inside_docker.html is that I do not need to mount the Docker socket volume.

I've set add_docker: true in my codeship-services.yml, which does the following:

Add the environment variables DOCKER_HOST, DOCKER_TLS_VERIFY and DOCKER_CERT_PATH from the host.
If DOCKER_CERT_PATH is set, it will mount the certificate directory through to the container. See add_docker for an example using Docker-in-Docker.

https://documentation.codeship.com/pro/builds-and-configuration/services/

@estebangarcia
Copy link

I'm having the same issue with CircleCI. Using v1.6.0

@kiview
Copy link
Member

kiview commented Mar 31, 2018

Hi @estebangarcia, would it be possible for you to migrate to CircleCI 2.0?
We are using CircleCI 2.0 in Testcontainers ourselves, so everything should work fine out of the box (look at our CircleCI config).

Since pre 2.0 CircleCI was using lxc directly themselves (AFAIK?), you have to jump through a lot more hoops to get Testcontainers working there.

@bsideup
Copy link
Member

bsideup commented Apr 11, 2018

Hi @gavinmh,
Could you please try with 1.7.0?

@kiview
Copy link
Member

kiview commented May 15, 2018

Since this is possibly fixed by the fix for #621 I'll close this for now.
@gavinmh I'll gladly reopen the issue if the problems persists for you on 1.7.0 🙂

@kszalontai
Copy link

I'm having the same issue with version 1.8.1
Codeship pro, add_docker=true, ryuk.container.timeout = 120

@sic2
Copy link

sic2 commented Aug 2, 2018

Yes, please. Re-open this issue since it still persists in 1.8.1

@kszalontai
Copy link

@kiview Could you please reopen this? ty

@kiview
Copy link
Member

kiview commented Aug 6, 2018

Reopened as requested, bug seems to be still present.

@kiview kiview reopened this Aug 6, 2018
@kiview kiview added the type/bug label Aug 6, 2018
@bountin
Copy link

bountin commented Aug 13, 2018

We're experiencing the same when the load on the build server (Jenkins) is (very) high.

One thing that eased the pain was upgrading docker where we used to use an early version of 2017. The docker service wasn't responsive during peak testing (eight parallel test runs) which lead me to that direction.

Still, we had to reduce the number of parallel builds because of testcontainer timeouts. I haven't tried to increase ryuk.container.timeout because IMO thirty seconds should be enough - also in a high load scenario.

@thammerl
Copy link
Contributor

@bountin I don't think the timeout is related to the issue we are having. I turned on the testcontainers logging and this is what I got:

14:56:23.689 INFO  [main] o.t.d.DockerClientProviderStrategy - [] - Loaded org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy from ~/.testcontainers.properties, will try it first
14:56:23.741 INFO  [main] o.t.d.DockerClientProviderStrategy - [] - Will use 'netty' transport
14:56:24.425 INFO  [main] o.t.d.EnvironmentAndSystemPropertyClientProviderStrategy - [] - Found docker client settings from environment
14:56:24.455 INFO  [main] o.t.d.DockerClientProviderStrategy - [] - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=unix:///var/run/docker.sock
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='jenkins'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=jenkins,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/var/lib/jenkins/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

14:56:24.458 INFO  [main] o.t.DockerClientFactory - [] - Docker host IP address is localhost
14:56:25.030 INFO  [main] o.t.DockerClientFactory - [] - Connected to docker: 
  Server Version: 18.03.1-ce
  API Version: 1.37
  Operating System: Ubuntu 16.04.5 LTS
  Total Memory: 20076 MB
14:56:26.727 WARN  [testcontainers-ryuk] o.t.u.ResourceReaper - [] - Can not connect to Ryuk at localhost:33333
java.net.SocketException: Broken pipe (Write failed)
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:134)
	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:144)
	at java.lang.Thread.run(Thread.java:748)
14:56:26.728 WARN  [testcontainers-ryuk] o.t.u.ResourceReaper - [] - Can not connect to Ryuk at localhost:33333
java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:210)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.readLine(BufferedReader.java:324)
	at java.io.BufferedReader.readLine(BufferedReader.java:389)
	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:147)
	at java.lang.Thread.run(Thread.java:748)

As you can see, only 1.7 seconds passed between connecting to docker and the socket exception.

@thammerl
Copy link
Contributor

thammerl commented Aug 20, 2018

After having a closer look at the code, I came up with the following findings/questions:

  1. Why are there two stack traces in the log output above? One when trying to write to the Ryuk socket and one when trying to read. Obviously there were two different threads trying to initialize the ryuk container. Shouldn't this only be executed once? Is the potential race condition in DockerClientFactory problematic? UPDATE: There are two stack traces because the index variable isn't incremented and the outermost while(true)-loop will try to connect again. As there aren't any further stack traces, I assume that the thread gets stuck in the while-loop waiting for the "ACK" response.
  2. Is it reasonable to just log a warning here? The ryukScheduledLatch will never be decremented to zero and will result in an IllegalStateException once the timeout is reached. Is it possible to recover from an IOException that is thrown when sending the filters to ryuk? I understand now that this retry-logic is intentional to wait until the ryuk server is available.
  3. It would be interesting to get the stdout of the ryuk server. Maybe it's possible to find the cause of the connection reset there. Anyone having an idea on how to do that?
  4. The sources JAR for 1.8.3 in Maven Central obviously does not contain the actual code of the release. I noticed when navigating the code in IDEA and realising that the line numbers in ResourceReaper from the stack trace don't match. The same is true for 1.8.2. Haven't checked any other releases. The sources for the releases on GitHub are fine.

Sorry if the answers to some of my questions are obvious. I'm new to the project and only guessing how things should play together here.

@thammerl
Copy link
Contributor

thammerl commented Aug 21, 2018

Now I had a build with numerous stack traces instead of just two. But after some attempts to connect it seems to get stuck in the while-loop waiting for the acknowledgment and after the 30 second timeout the IllegalStateException kicks in:

testcontainers.log

@thammerl
Copy link
Contributor

thammerl commented Aug 21, 2018

I've created a pull request that should fix the issue. See #843.

@vpavic
Copy link
Contributor

vpavic commented Aug 27, 2018

We're also occasionally experiencing the same issue with Spring Session builds on Jenkins. Here's the stacktrace from one of those builds:

java.lang.IllegalStateException: Failed to load ApplicationContext
	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:125)
	at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:108)
	at org.springframework.test.context.web.ServletTestExecutionListener.setUpRequestContextIfNecessary(ServletTestExecutionListener.java:190)
	at org.springframework.test.context.web.ServletTestExecutionListener.prepareTestInstance(ServletTestExecutionListener.java:132)
	at org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:246)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:227)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:289)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:291)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:246)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
	at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:117)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:155)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:137)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'indexController': Unsatisfied dependency expressed through field 'sessions'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.boot.autoconfigure.session.RedisSessionConfiguration$SpringBootRedisHttpSessionConfiguration': Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisConnectionFactory' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory]: Factory method 'redisConnectionFactory' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisContainer' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.containers.GenericContainer]: Factory method 'redisContainer' threw exception; nested exception is java.lang.IllegalStateException: Can not connect to Ryuk
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:598)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:90)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:376)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1376)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:575)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:829)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:865)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:548)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:785)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:418)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:337)
	at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:139)
	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99)
	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:117)
	... 49 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.boot.autoconfigure.session.RedisSessionConfiguration$SpringBootRedisHttpSessionConfiguration': Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisConnectionFactory' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory]: Factory method 'redisConnectionFactory' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisContainer' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.containers.GenericContainer]: Factory method 'redisContainer' threw exception; nested exception is java.lang.IllegalStateException: Can not connect to Ryuk
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:584)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:392)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1282)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1126)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:538)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:290)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1222)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1149)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:595)
	... 67 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisConnectionFactory' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory]: Factory method 'redisConnectionFactory' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisContainer' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.containers.GenericContainer]: Factory method 'redisContainer' threw exception; nested exception is java.lang.IllegalStateException: Can not connect to Ryuk
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:624)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:455)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1282)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1126)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:538)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:290)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1222)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.getObject(DefaultListableBeanFactory.java:1775)
	at org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration.setRedisConnectionFactory(RedisHttpSessionConfiguration.java:181)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:710)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:90)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:376)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1376)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:575)
	... 85 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory]: Factory method 'redisConnectionFactory' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisContainer' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.containers.GenericContainer]: Factory method 'redisContainer' threw exception; nested exception is java.lang.IllegalStateException: Can not connect to Ryuk
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:619)
	... 107 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisContainer' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.containers.GenericContainer]: Factory method 'redisContainer' threw exception; nested exception is java.lang.IllegalStateException: Can not connect to Ryuk
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:624)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:455)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1282)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1126)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:538)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.resolveBeanReference(ConfigurationClassEnhancer.java:394)
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:366)
	at sample.FindByUsernameTests$Config$$EnhancerBySpringCGLIB$$5030775a.redisContainer(<generated>)
	at sample.FindByUsernameTests$Config.redisConnectionFactory(FindByUsernameTests.java:95)
	at sample.FindByUsernameTests$Config$$EnhancerBySpringCGLIB$$5030775a.CGLIB$redisConnectionFactory$1(<generated>)
	at sample.FindByUsernameTests$Config$$EnhancerBySpringCGLIB$$5030775a$$FastClassBySpringCGLIB$$e4d393ce.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)
	at sample.FindByUsernameTests$Config$$EnhancerBySpringCGLIB$$5030775a.redisConnectionFactory(<generated>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	... 108 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.containers.GenericContainer]: Factory method 'redisContainer' threw exception; nested exception is java.lang.IllegalStateException: Can not connect to Ryuk
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:619)
	... 131 more
Caused by: java.lang.IllegalStateException: Can not connect to Ryuk
	at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:166)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:119)
	at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:151)
	at sample.FindByUsernameTests$Config.redisContainer(FindByUsernameTests.java:87)
	at sample.FindByUsernameTests$Config$$EnhancerBySpringCGLIB$$5030775a.CGLIB$redisContainer$0(<generated>)
	at sample.FindByUsernameTests$Config$$EnhancerBySpringCGLIB$$5030775a$$FastClassBySpringCGLIB$$e4d393ce.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)
	at sample.FindByUsernameTests$Config$$EnhancerBySpringCGLIB$$5030775a.redisContainer(<generated>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	... 132 more

@kiview
Copy link
Member

kiview commented Oct 1, 2018

@vpavic Does this still happen for you, or was it resolved with #843 in 1.9.0?

@vpavic
Copy link
Contributor

vpavic commented Oct 1, 2018

Thanks for the ping - I forgot to follow up on this.

We were still experiencing the problem with 1.9.0-rc1, and have set ryuk.container.timeout=120 property since. This appears to have fixed it, but we've also had some updated on this infrastructure side and have upgrade to 1.9.1 since.

I'll try to remove ryuk.container.timeout=120 some time this week and see how it behaves, just to try
and isolate the problem.

@guss77
Copy link
Contributor

guss77 commented Oct 29, 2018

I'm having the same issue with 1.9.1 on Circle CI (2.0). I'm running testcontainers for a JDBC server for Jooq code generation:

[INFO] Will use 'okhttp' transport
[INFO] Found docker client settings from environment
[INFO] Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=tcp://35.231.185.114:2376
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='circleci'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=tcp://35.231.185.114:2376,registryUsername=circleci,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/home/circleci/.docker,sslConfig=LocalDirectorySSLConfig{dockerCertPath=/tmp/docker-certs194934439},apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

[INFO] Docker host IP address is 35.231.185.114
[INFO] Connected to docker: 
  Server Version: 17.09.0-ce
  API Version: 1.32
  Operating System: Ubuntu 16.04.3 LTS
  Total Memory: 7477 MB
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:42 min
[INFO] Finished at: 2018-10-29T07:45:45Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.jooq:jooq-codegen-maven:3.11.5:generate (default) on project model: Error running jOOQ code generation tool: Can not connect to Ryuk -> [Help 1]

I can't figure out how to set ryuk.container.timeout=120 - usind -D on the maven command line doesn't seem to do anything.

Here's the full debug log when running Maven with -X:

[DEBUG] Container not found in cache, creating new instance
[DEBUG] Testcontainers configuration overrides will be loaded from file:/home/circleci/.testcontainers.properties
[INFO] Will use 'okhttp' transport
[DEBUG] Pinging docker daemon...
[DEBUG] Cmd: org.testcontainers.dockerclient.transport.okhttp.OkHttpDockerCmdExecFactory$1@34349e19
[INFO] Found docker client settings from environment
[INFO] Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=tcp://35.190.177.84:2376
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='circleci'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=tcp://35.190.177.84:2376,registryUsername=circleci,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/home/circleci/.docker,sslConfig=LocalDirectorySSLConfig{dockerCertPath=/tmp/docker-certs196266953},apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

[INFO] Docker host IP address is 35.190.177.84
[DEBUG] Cmd: com.github.dockerjava.core.exec.InfoCmdExec@986b619
[DEBUG] Cmd: com.github.dockerjava.core.exec.VersionCmdExec@7df3da0b
[INFO] Connected to docker: 
  Server Version: 17.09.0-ce
  API Version: 1.32
  Operating System: Ubuntu 16.04.3 LTS
  Total Memory: 7477 MB
[DEBUG] Cmd: ListImagesCmdImpl[imageNameFilter=quay.io/testcontainers/ryuk:0.2.2,showAll=false,filters=com.github.dockerjava.core.util.FiltersBuilder@0,execution=com.github.dockerjava.core.exec.ListImagesCmdExec@3514ac7d]
[DEBUG] Looking up auth config for image: quay.io/testcontainers/ryuk:0.2.2
[DEBUG] RegistryAuthLocator has configFile: /home/circleci/.docker/config.json (exists) and commandPathPrefix: 
[DEBUG] registryName [quay.io] for dockerImageName [quay.io/testcontainers/ryuk:0.2.2]
[DEBUG] no matching Auth Configs - falling back to defaultAuthConfig [null]
[DEBUG] Effective auth config [null]
[DEBUG] PullResponseItem[stream=<null>,status=Pulling from testcontainers/ryuk,progressDetail=<null>,progress=<null>,id=0.2.2,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Pulling fs layer,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Pulling fs layer,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=c9ea39c9283a,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Pulling fs layer,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Downloading,progressDetail=ResponseItem.ProgressDetail[current=16384,total=308109,start=<null>],progress=[==>                                                ]  16.38kB/308.1kB,id=c9ea39c9283a,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Downloading,progressDetail=ResponseItem.ProgressDetail[current=32768,total=1776708,start=<null>],progress=[>                                                  ]  32.77kB/1.777MB,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Downloading,progressDetail=ResponseItem.ProgressDetail[current=32279,total=2065537,start=<null>],progress=[>                                                  ]  32.28kB/2.066MB,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Download complete,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=c9ea39c9283a,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Downloading,progressDetail=ResponseItem.ProgressDetail[current=655360,total=1776708,start=<null>],progress=[==================>                                ]  655.4kB/1.777MB,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Downloading,progressDetail=ResponseItem.ProgressDetail[current=588961,total=2065537,start=<null>],progress=[==============>                                    ]    589kB/2.066MB,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Verifying Checksum,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Download complete,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Verifying Checksum,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Download complete,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=32768,total=2065537,start=<null>],progress=[>                                                  ]  32.77kB/2.066MB,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=1605632,total=2065537,start=<null>],progress=[======================================>            ]  1.606MB/2.066MB,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=2065537,total=2065537,start=<null>],progress=[==================================================>]  2.066MB/2.066MB,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Pull complete,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=32768,total=308109,start=<null>],progress=[=====>                                             ]  32.77kB/308.1kB,id=c9ea39c9283a,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=308109,total=308109,start=<null>],progress=[==================================================>]  308.1kB/308.1kB,id=c9ea39c9283a,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=308109,total=308109,start=<null>],progress=[==================================================>]  308.1kB/308.1kB,id=c9ea39c9283a,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Pull complete,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=c9ea39c9283a,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=32768,total=1776708,start=<null>],progress=[>                                                  ]  32.77kB/1.777MB,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=1146880,total=1776708,start=<null>],progress=[================================>                  ]  1.147MB/1.777MB,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=1776708,total=1776708,start=<null>],progress=[==================================================>]  1.777MB/1.777MB,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Pull complete,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Digest: sha256:3d53f437c84a355eafccfef4a0fe3270c49893bd2ac80b5ef57698c1a9102fec,progressDetail=<null>,progress=<null>,id=<null>,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Status: Downloaded newer image for quay.io/testcontainers/ryuk:0.2.2,progressDetail=<null>,progress=<null>,id=<null>,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] Looking up auth config for image: quay.io/testcontainers/ryuk:0.2.2
[DEBUG] RegistryAuthLocator has configFile: /home/circleci/.docker/config.json (exists) and commandPathPrefix: 
[DEBUG] registryName [quay.io] for dockerImageName [quay.io/testcontainers/ryuk:0.2.2]
[DEBUG] no matching Auth Configs - falling back to defaultAuthConfig [null]
[DEBUG] Effective auth config [null]
[DEBUG] Cmd: com.github.dockerjava.core.command.CreateContainerCmdImpl@fd5689d[name=testcontainers-ryuk-b1afc90d-70e5-418e-b741-ed3433252737,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd=<null>,entrypoint=<null>,image=quay.io/testcontainers/ryuk:0.2.2,volumes=com.github.dockerjava.api.model.Volumes@1d9af731,workingDir=<null>,macAddress=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@39afe59f,stopSignal=<null>,hostConfig=com.github.dockerjava.api.model.HostConfig@16f34376[binds=com.github.dockerjava.api.model.Binds@4272d8cf,blkioWeight=<null>,blkioWeightDevice=<null>,blkioDeviceReadBps=<null>,blkioDeviceReadIOps=<null>,blkioDeviceWriteBps=<null>,blkioDeviceWriteIOps=<null>,memorySwappiness=<null>,capAdd=<null>,capDrop=<null>,containerIDFile=<null>,cpuPeriod=<null>,cpuShares=<null>,cpuQuota=<null>,cpusetCpus=<null>,cpusetMems=<null>,devices=<null>,diskQuota=<null>,dns=<null>,dnsSearch=<null>,extraHosts=<null>,links=<null>,logConfig=<null>,lxcConf=<null>,memory=<null>,memorySwap=<null>,memoryReservation=<null>,kernelMemory=<null>,networkMode=<null>,oomKillDisable=<null>,autoRemove=true,oomScoreAdj=<null>,portBindings=<null>,privileged=<null>,publishAllPorts=true,readonlyRootfs=<null>,restartPolicy=<null>,ulimits=<null>,volumesFrom=<null>,pidMode=<null>,securityOpts=<null>,cgroupParent=<null>,volumeDriver=<null>,shmSize=<null>,pidsLimit=<null>,runtime=<null>,tmpFs=<null>],labels={org.testcontainers=true},networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=<null>,execution=com.github.dockerjava.core.exec.CreateContainerCmdExec@4e96cb04]
[DEBUG] Cmd: 5905dfcfefceb40da10b551e3c8b9c51e5618001ef9d19683eb844eef70b7352,com.github.dockerjava.core.exec.StartContainerCmdExec@2b974137
[DEBUG] Cmd: 5905dfcfefceb40da10b551e3c8b9c51e5618001ef9d19683eb844eef70b7352,false,com.github.dockerjava.core.exec.InspectContainerCmdExec@76fc5687
[DEBUG] GET: OkHttpWebTarget(okHttpClient=org.testcontainers.shaded.okhttp3.OkHttpClient@53fbb2b4, baseUrl=https://35.190.177.84:2376/, path=[/containers/5905dfcfefceb40da10b551e3c8b9c51e5618001ef9d19683eb844eef70b7352/json], queryParams={})
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:48 min
[INFO] Finished at: 2018-10-29T07:51:37Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.jooq:jooq-codegen-maven:3.11.5:generate (default) on project model: Error running jOOQ code generation tool: Can not connect to Ryuk -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.jooq:jooq-codegen-maven:3.11.5:generate (default) on project model: Error running jOOQ code generation tool
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Error running jOOQ code generation tool
    at org.jooq.codegen.maven.Plugin.execute (Plugin.java:200)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: java.lang.IllegalStateException: Can not connect to Ryuk
    at org.testcontainers.utility.ResourceReaper.start (ResourceReaper.java:148)
    at org.testcontainers.utility.ResourceReaper.start (ResourceReaper.java:65)
    at org.testcontainers.DockerClientFactory.client (DockerClientFactory.java:124)
    at org.testcontainers.containers.GenericContainer.<init> (GenericContainer.java:162)
    at org.testcontainers.containers.JdbcDatabaseContainer.<init> (JdbcDatabaseContainer.java:40)
    at org.testcontainers.containers.MariaDBContainer.<init> (MariaDBContainer.java:25)
    at org.testcontainers.containers.MariaDBContainerProvider.newInstance (MariaDBContainerProvider.java:19)
    at org.testcontainers.containers.MariaDBContainerProvider.newInstance (MariaDBContainerProvider.java:14)
    at org.testcontainers.containers.JdbcDatabaseContainerProvider.newInstance (JdbcDatabaseContainerProvider.java:49)
    at org.testcontainers.jdbc.ContainerDatabaseDriver.connect (ContainerDatabaseDriver.java:94)
    at org.jooq.codegen.GenerationTool.run (GenerationTool.java:318)
    at org.jooq.codegen.GenerationTool.generate (GenerationTool.java:220)
    at org.jooq.codegen.maven.Plugin.execute (Plugin.java:197)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

@vpavic
Copy link
Contributor

vpavic commented Oct 30, 2018

@guss See the docs on custom properties.

As a side note for @kiview and @bsideup, it would be nice to have ryuk.container.timeout (or any other missing properties) added to that page for a complete reference.

@guss77
Copy link
Contributor

guss77 commented Oct 30, 2018

Sorry about the weird link, Github is sometimes too enthusiastic about referencing issues.

I've tried with setting the timeout to 120, and indeed the build takes 2 more minutes but the Ryuk connection still fails:

[DEBUG] PullResponseItem[stream=<null>,status=Status: Downloaded newer image for quay.io/testcontainers/ryuk:0.2.2,progressDetail=<null>,progress=<null>,id=<null>,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] Looking up auth config for image: quay.io/testcontainers/ryuk:0.2.2
[DEBUG] RegistryAuthLocator has configFile: /home/circleci/.docker/config.json (exists) and commandPathPrefix: 
[DEBUG] registryName [quay.io] for dockerImageName [quay.io/testcontainers/ryuk:0.2.2]
[DEBUG] no matching Auth Configs - falling back to defaultAuthConfig [null]
[DEBUG] Effective auth config [null]
[DEBUG] Cmd: com.github.dockerjava.core.command.CreateContainerCmdImpl@6ba0ee4a[name=testcontainers-ryuk-0124f49d-4059-439a-93b6-cb8f82b9ac4e,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd=<null>,entrypoint=<null>,image=quay.io/testcontainers/ryuk:0.2.2,volumes=com.github.dockerjava.api.model.Volumes@27db45f,workingDir=<null>,macAddress=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@6ec3d8e4,stopSignal=<null>,hostConfig=com.github.dockerjava.api.model.HostConfig@1aeff8ca[binds=com.github.dockerjava.api.model.Binds@6068ebb2,blkioWeight=<null>,blkioWeightDevice=<null>,blkioDeviceReadBps=<null>,blkioDeviceReadIOps=<null>,blkioDeviceWriteBps=<null>,blkioDeviceWriteIOps=<null>,memorySwappiness=<null>,capAdd=<null>,capDrop=<null>,containerIDFile=<null>,cpuPeriod=<null>,cpuShares=<null>,cpuQuota=<null>,cpusetCpus=<null>,cpusetMems=<null>,devices=<null>,diskQuota=<null>,dns=<null>,dnsSearch=<null>,extraHosts=<null>,links=<null>,logConfig=<null>,lxcConf=<null>,memory=<null>,memorySwap=<null>,memoryReservation=<null>,kernelMemory=<null>,networkMode=<null>,oomKillDisable=<null>,autoRemove=true,oomScoreAdj=<null>,portBindings=<null>,privileged=<null>,publishAllPorts=true,readonlyRootfs=<null>,restartPolicy=<null>,ulimits=<null>,volumesFrom=<null>,pidMode=<null>,securityOpts=<null>,cgroupParent=<null>,volumeDriver=<null>,shmSize=<null>,pidsLimit=<null>,runtime=<null>,tmpFs=<null>],labels={org.testcontainers=true},networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=<null>,execution=com.github.dockerjava.core.exec.CreateContainerCmdExec@5a466dd]
[DEBUG] Cmd: ee8c93190996a894ee8eb46d2aa2435b81b6c902cec44fdc831c019453cbd73a,com.github.dockerjava.core.exec.StartContainerCmdExec@4aac81ca
[DEBUG] Cmd: ee8c93190996a894ee8eb46d2aa2435b81b6c902cec44fdc831c019453cbd73a,false,com.github.dockerjava.core.exec.InspectContainerCmdExec@269c7104
[DEBUG] GET: OkHttpWebTarget(okHttpClient=org.testcontainers.shaded.okhttp3.OkHttpClient@6de84336, baseUrl=https://35.185.49.118:2376/, path=[/containers/ee8c93190996a894ee8eb46d2aa2435b81b6c902cec44fdc831c019453cbd73a/json], queryParams={})
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:23 min
[INFO] Finished at: 2018-10-30T11:38:48Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.jooq:jooq-codegen-maven:3.11.5:generate (default) on project model: Error running jOOQ code generation tool: Can not connect to Ryuk -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.jooq:jooq-codegen-maven:3.11.5:generate (default) on project model: Error running jOOQ code generation tool
...
Caused by: java.lang.IllegalStateException: Can not connect to Ryuk
    at org.testcontainers.utility.ResourceReaper.start (ResourceReaper.java:148)
    at org.testcontainers.utility.ResourceReaper.start (ResourceReaper.java:65)
    at org.testcontainers.DockerClientFactory.client (DockerClientFactory.java:124)
    at org.testcontainers.containers.GenericContainer.<init> (GenericContainer.java:162)
    at org.testcontainers.containers.JdbcDatabaseContainer.<init> (JdbcDatabaseContainer.java:40)
    at org.testcontainers.containers.MariaDBContainer.<init> (MariaDBContainer.java:25)
...

@rafaelalmeida
Copy link

@guss77 Did you manage to solve this? Running into the same issue now with 1.10.2.

@ysb33r
Copy link

ysb33r commented Feb 6, 2019

I am seeing the same issue with 1.10.6 but only on Mac running Docker Desktop

prent

Similar stack trace:

    java.lang.IllegalStateException: Can not connect to Ryuk
        at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:148)
        at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:125)
        at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:142)

I am running this side-by-side with a Linux laptop (Docker engine 18.06.0-dev build e68fc7a) that has less memory than the Mac, but I am not seeing the issue there.

@kiview
Copy link
Member

kiview commented Feb 7, 2019

@ysb33r Did you already try to increase the timeout in ryuk.container.timeout?
https://www.testcontainers.org/features/configuration/

@ysb33r
Copy link

ysb33r commented Feb 7, 2019

Did you already try to increase the timeout in ryuk.container.timeout?
https://www.testcontainers.org/features/configuration/

Yes I have set it to 120, but possibly a complete restart of Docker Desktop seemed to have resolved the problem.

@ciri-cuervo
Copy link

ciri-cuervo commented Apr 15, 2019

have tried creating the docker group? it solves permission issues

# create new group
sudo groupadd docker
# add current user to the group
sudo usermod -aG docker $USER
# log in to the group
newgrp - docker

@AlexanderArgyriou
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests