Reproducible "IllegalStateException: Can not connect to Ryuk" (on Fedora 29 with "docker" 1.13.1 - runc)

[vorburger]

I am hitting the IllegalStateException: Can not connect to Ryuk problem similar to #581, #780, #892, #1014 et al, but in my case it's not sporadic but always 100% reproducible - see vorburger/opendaylight-etcd#14.

It also happens with a much simpler reproducer project like this:

The pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>ch.vorburger</groupId>
  <artifactId>testcontainer-quicktest</artifactId>
  <version>1.0.0-SNAPSHOT</version>

  <dependencies>
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>mariadb</artifactId>
      <version>1.11.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.12</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-simple</artifactId>
      <version>1.7.26</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
</project>

and src/test/java/ch/vorburger/testcontainer/quicktest/TheTest.java :

package ch.vorburger.testcontainer.quicktest;

import org.junit.Rule;
import org.junit.Test;
import org.testcontainers.containers.MariaDBContainer;

public class TheTest {

    @Rule
    public MariaDBContainer mariaDB = new MariaDBContainer();

    @Test
    public void someTestMethod() {
        String url = mariaDB.getJdbcUrl();
    }
}

As mentioned in vorburger/opendaylight-etcd#14, this is probably of interest & related:

$ systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-03-27 20:07:31 CET; 1 weeks 0 days ago
     Docs: http://docs.docker.com
 Main PID: 1336 (dockerd-current)
    Tasks: 19 (limit: 8192)
   Memory: 310.0M
   CGroup: /system.slice/docker.service
           └─1336 /usr/bin/dockerd-current --add-runtime oci=/usr/libexec/docker/docker-runc-current --default-runtime=oci --authorization-plugin>

Apr 04 12:25:40 khany dockerd-current[1336]: time="2019-04-04T12:25:40.006299612+02:00" level=warning msg="bb8dd354bc44de4dd8a8ea73373da94f8541f8>
Apr 04 12:27:03 khany dockerd-current[1336]: time="2019-04-04T12:27:03.278689640+02:00" level=warning msg="failed to retrieve docker-runc version>
Apr 04 12:27:03 khany dockerd-current[1336]: time="2019-04-04T12:27:03.279152622+02:00" level=warning msg="failed to retrieve docker-init version>
Apr 04 12:27:04 khany dockerd-current[1336]: 2019/04/04 10:27:04 Starting on port 8080...
Apr 04 12:27:04 khany dockerd-current[1336]: panic: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/>
Apr 04 12:27:04 khany dockerd-current[1336]: 
Apr 04 12:27:04 khany dockerd-current[1336]: goroutine 1 [running]:
Apr 04 12:27:04 khany dockerd-current[1336]: main.main()
Apr 04 12:27:04 khany dockerd-current[1336]:         /go/src/github.com/testcontainers/moby-ryuk/main.go:31 +0xe2e
Apr 04 12:27:04 khany dockerd-current[1336]: time="2019-04-04T12:27:04.452543464+02:00" level=warning msg="b1a29a551b79fb1cada802cfe0543a4daba9db>

As mentioned in vorburger/opendaylight-etcd#14, disabling Ryuk by setting TESTCONTAINERS_RYUK_DISABLED environment variable to true works around it (but in an ideal world, this would not be required and the root cause of this fixed).

This is on Fedora 29 with "docker" 1.13.1 from Fedora package docker-2:1.13.1-65.git1185cfd.fc29.x86_64, not docker.com:

$ docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-65.git1185cfd.fc29.x86_64
 Go version:      go1.11.5
 Git commit:      1185cfd/1.13.1
 Built:           Tue Feb 12 17:46:58 2019
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-65.git1185cfd.fc29.x86_64
 Go version:      go1.11.5
 Git commit:      1185cfd/1.13.1
 Built:           Tue Feb 12 17:46:58 2019
 OS/Arch:         linux/amd64
 Experimental:    false

[vorburger]

BTW: Just building this project on the same machine hits a similar problem:

$ ./gradlew build

> Task :neo4j:test

Gradle Test Executor 1 > org.testcontainers.containers.Neo4jContainerJUnitIntegrationTest > classMethod STARTED

Gradle Test Executor 1 > org.testcontainers.containers.Neo4jContainerJUnitIntegrationTest > classMethod FAILED
    java.lang.ExceptionInInitializerError

        Caused by:
        java.lang.IllegalStateException: Can not connect to Ryuk
            at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:148)
            at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:125)
            at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:138)
            at org.testcontainers.containers.Neo4jContainer.<init>(Neo4jContainer.java:78)
            at org.testcontainers.containers.Neo4jContainer.<init>(Neo4jContainer.java:67)
            at org.testcontainers.containers.Neo4jContainerJUnitIntegrationTest.<clinit>(Neo4jContainerJUnitIntegrationTest.java:22)

Gradle Test Executor 1 > org.testcontainers.containers.Neo4jContainerTest > shouldRunEnterprise STARTED

Gradle Test Executor 1 > org.testcontainers.containers.Neo4jContainerTest > shouldRunEnterprise SKIPPED

Gradle Test Executor 1 > org.testcontainers.containers.Neo4jContainerTest > shouldCopyPlugins STARTED
<=========----> 70% EXECUTING [26m 59s]
> :neo4j:test > 2 tests completed, 1 failed, 1 skipped
> :neo4j:test > Executing test org.testcontainers.containers.Neo4jContainerTest

[kiview]

This is a really old Docker version (1.13 was release in January 2017) and I always wonder what distributions are doing when building their Docker packages in general.

Testcontainers on Fedora 29 with Docker from Docker CE repository works fine:

➜  ~ docker version
Client:
 Version:           18.09.4
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        d14af54
 Built:             Wed Mar 27 18:36:04 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.4
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       d14af54
  Built:            Wed Mar 27 18:04:46 2019
  OS/Arch:          linux/amd64
  Experimental:     false

I know our docs say that Testcontainers requires Docker version 1.10 on Linux, but I don't think Docker backward compatibility is something we actively ensure.

However, this could also be something related to SELinux config for the Docker package in Fedora?

[vorburger]

I can confirm that (first un-installing and then) installing Docker.com CE via https://docs.docker.com/install/linux/docker-ce/fedora/, instead of Fedora's Docker, works around this problem and makes Testcontainers "just work" out of the box.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

[stale]

This issue has been automatically closed due to inactivity. We apologise if this is still an active problem for you, and would ask you to re-open the issue if this is the case.