running example job: failed to create container from image redis:latest: no such image

[mcuadros]

When trying to create a job example using the nomad init I get the following error:
failed to create container from image redis:latest: no such image

The image was correctly pulled since making docker images, redis:latest can be found.

I am running 3 servers and 2 nodes inside of Docker in GCE, all setup using terraform

Nomad version

Nomad 0.3.1

Operating system and Environment details

CoreOS 983.0.0 (Coeur Rouge)
Running inside Docker 1.10.2

Nomad Client logs

  Loaded configuration from /etc/client.hcl
==> Starting Nomad agent...
==> Nomad agent configuration:

                 Atlas: <disabled>
                Client: true
             Log Level: DEBUG
                Region: europe-west1-d (DC: srcd-test)

Server: false

==> Nomad agent started! Log data will stream in below:

    2016/04/12 18:09:44 [INFO] client: using state directory /data/nomad/client
    2016/04/12 18:09:44 [INFO] client: using alloc directory /data/nomad/alloc
    2016/04/12 18:09:44 [INFO] fingerprint.cgroups: cgroups are available
    2016/04/12 18:09:44 [DEBUG] client: periodically fingerprinting cgroup at duration 15s
    2016/04/12 18:09:44 [DEBUG] client: periodically fingerprinting consul at duration 15s
    2016/04/12 18:09:44 [DEBUG] fingerprint.network: Detected interface eth0  with IP 172.17.0.4 during fingerprinting
    2016/04/12 18:09:44 [DEBUG] client: applied fingerprints [arch cgroup cpu env_gce host memory network storage]
    2016/04/12 18:09:44 [DEBUG] driver.docker: using client connection initialized from environment
    2016/04/12 18:09:44 [DEBUG] driver.docker: privileged containers are disabled
    2016/04/12 18:09:44 [DEBUG] client: available drivers [docker exec]
    2016/04/12 18:09:44 [INFO] client: setting server address list: [nomad.service.srcd:4647]
    2016/04/12 18:09:44 [DEBUG] client: periodically checking for node changes at duration 5s
    2016/04/12 18:09:44 [DEBUG] client: updated allocations at index 167 (pulled 0) (filtered 0)
    2016/04/12 18:09:44 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 0)
    2016/04/12 18:09:44 [DEBUG] client: node registration complete
    2016/04/12 18:09:49 [DEBUG] client: state changed, updating node.
    2016/04/12 18:09:49 [DEBUG] client: node registration complete
    2016/04/12 18:11:54 [DEBUG] client: updated allocations at index 184 (pulled 0) (filtered 0)
    2016/04/12 18:11:54 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 0)
    2016/04/12 18:11:54 [DEBUG] client: updated allocations at index 190 (pulled 1) (filtered 0)
    2016/04/12 18:11:54 [DEBUG] client: allocs: (added 1) (removed 0) (updated 0) (ignore 0)
    2016/04/12 18:11:54 [DEBUG] client: starting task runners for alloc '0094a3d5-dab9-1460-d9d1-0847c43cfc30'
    2016/04/12 18:11:54 [DEBUG] client: starting task context for 'redis' (alloc '0094a3d5-dab9-1460-d9d1-0847c43cfc30')
    2016/04/12 18:12:08 [DEBUG] driver.docker: docker pull redis:latest succeeded
    2016/04/12 18:12:08 [DEBUG] driver.docker: identified image redis:latest as sha256:4f5f397d4b7ca414891bd2959ef71c83bb7010d095efb2497f0b2f407cb50f0d
    2016/04/12 18:12:08 [DEBUG] plugin: starting plugin: /usr/bin/nomad []string{"/usr/bin/nomad", "syslog", "/data/nomad/alloc/0094a3d5-dab9-1460-d9d1-0847c43cfc30/redis/redis-syslog-collector.out"}
    2016/04/12 18:12:08 [DEBUG] plugin: waiting for RPC address for: /usr/bin/nomad
    2016/04/12 18:12:08 [DEBUG] plugin: nomad: 2016/04/12 18:12:08 [DEBUG] plugin: plugin address: unix /tmp/plugin914362017
    2016/04/12 18:12:08 [DEBUG] driver.docker: using 268435456 bytes memory for redis:latest
    2016/04/12 18:12:08 [DEBUG] driver.docker: using 500 cpu shares for redis:latest
    2016/04/12 18:12:08 [DEBUG] driver.docker: binding directories []string{"/data/nomad/alloc/0094a3d5-dab9-1460-d9d1-0847c43cfc30/alloc:/alloc:rw,z", "/data/nomad/alloc/0094a3d5-dab9-1460-d9d1-0847c43cfc30/redis:/local:rw,Z"} for redis:latest
    2016/04/12 18:12:08 [DEBUG] driver.docker: networking mode not specified; defaulting to bridge
    2016/04/12 18:12:08 [DEBUG] driver.docker: allocated port 172.17.0.4:23904 -> 6379 (mapped)
    2016/04/12 18:12:08 [DEBUG] driver.docker: exposed port 6379
    2016/04/12 18:12:08 [DEBUG] driver.docker: setting container name to: redis-0094a3d5-dab9-1460-d9d1-0847c43cfc30
    2016/04/12 18:12:08 [ERR] driver.docker: failed to create container from image redis:latest: no such image
    2016/04/12 18:12:08 [DEBUG] plugin: /usr/bin/nomad: plugin process exited
    2016/04/12 18:12:08 [ERR] client: failed to start task 'redis' for alloc '0094a3d5-dab9-1460-d9d1-0847c43cfc30': Failed to create container from image redis:latest: no such image

Reproduction steps

client config:

      bind_addr = "0.0.0.0"
      data_dir = "/data/nomad"

      client {
        enabled = true
        node_class = "${kind}"
        servers = ["nomad.service.srcd:4647"]
        options {
          consul.address = "172.17.0.1:8500"
        }
      }

server config:

      bind_addr = "0.0.0.0"
      data_dir = "/data/nomad"

      advertise {
        rpc = "$private_ipv4:4647"
        serf = "$private_ipv4:4648"
      }

      server {
        enabled = true
        bootstrap_expect = ${size}
        retry_join = ["nomad.service.srcd"]
      }

Job file (if appropriate)

Default one, changing datacenter value

[dadgar]

Does this happen randomly or every time for you?

[mcuadros]

everytime, and in all the nodes.

[dadgar]

Ah I didn't see that you are running it in a docker container. We don't really test for that use case currently so you are in a bit of uncharted territory. I would suggest that if you can, run Nomad outside of the container. Nomad client itself is acting as a supervisor and resource isolator.

[mcuadros]

If nomad is connecting to the socket, should be no problem. Our policy enforce us to run any service under docker. We are testing Swarm and Kubernetes and all is running in docker containers.

[ksinica]

I've had similar problem. Sharing /tmp directory between host and a container fixed it. Docker service was exepcting to find the plugin file in host /tmp directory, but the file was created in the container filesystem:

POST /containers/c7cc6fdca7431507136a8ed06a2b6d04f620aaf5dd0dc52e177106c24a093239/start HTTP/1.1\r
Host: unix.sock\r
User-Agent: go-dockerclient\r
Content-Length: 4\r
Content-Type: application/json\r
Accept-Encoding: gzip\r
\r
null

HTTP/1.1 404 Not Found\r
Content-Type: text/plain; charset=utf-8\r
Server: Docker/1.10.3 (linux)\r
X-Content-Type-Options: nosniff\r
Date: Tue, 26 Apr 2016 14:44:53 GMT\r
Content-Length: 53\r
\r
stat /tmp/plugin155390308: no such file or directory

[dadgar]

@mcuadros Do you want to try @ksinica suggestion and see if it solved your problem?

[joeshaw]

The "no such image" error is a red herring, caused by a bug in go-dockerclient: fsouza/go-dockerclient#528

[chuyskywalker]

Adding a -v /tmp:/tmp to my container invocaction fixed this for me.

[sethvargo]

Hey @dadgar,

I'm hitting this as well in the "happy path" to getting started with Nomad on Mac with the native docker client. First, I hit #1743, but that was an easy fix.

Now, even though I am running Docker for mac and have the redis image downloaded, Nomad is still unable to find the image. I tried mounting the /tmp directory as suggested, but that does not work either.

config {
  image = "redis:latest"
  port_map {
    db = 6379
  }
  args = ["-v", "/tmp:/tmp"]
}

2016/09/24 14:16:06 [DEBUG] plugin: waiting for RPC address for: /Users/sethvargo/Development/go/bin/nomad
    2016/09/24 14:16:06 [DEBUG] plugin: nomad: 2016/09/24 14:16:06 [DEBUG] plugin: plugin address: unix /var/folders/hq/mnms8w455qxgfvbcw7thydm40000gn/T/plugin671827800
    2016/09/24 14:16:06.306949 [DEBUG] driver.docker: using 268435456 bytes memory for redis
    2016/09/24 14:16:06.306972 [DEBUG] driver.docker: using 500 cpu shares for redis
    2016/09/24 14:16:06.306994 [DEBUG] driver.docker: binding directories []string{"/var/folders/hq/mnms8w455qxgfvbcw7thydm40000gn/T/NomadClient915821458/dec0e1e4-88a7-9951-d30f-9c0ff52715e7/alloc:/alloc", "/var/folders/hq/mnms8w455qxgfvbcw7thydm40000gn/T/NomadClient915821458/dec0e1e4-88a7-9951-d30f-9c0ff52715e7/redis:/local", "/var/folders/hq/mnms8w455qxgfvbcw7thydm40000gn/T/NomadClient915821458/dec0e1e4-88a7-9951-d30f-9c0ff52715e7/redis/secrets:/secrets"} for redis
    2016/09/24 14:16:06.307007 [DEBUG] driver.docker: networking mode not specified; defaulting to bridge
    2016/09/24 14:16:06.307019 [DEBUG] driver.docker: allocated port 127.0.0.1:43925 -> 6379 (mapped)
    2016/09/24 14:16:06.307026 [DEBUG] driver.docker: exposed port 6379
    2016/09/24 14:16:06.307099 [DEBUG] driver.docker: setting container name to: redis-dec0e1e4-88a7-9951-d30f-9c0ff52715e7
    2016/09/24 14:16:06.314572 [ERR] driver.docker: failed to create container from image redis:latest: no such image
    2016/09/24 14:16:06 [DEBUG] plugin: /Users/sethvargo/Development/go/bin/nomad: plugin process exited

FWIW, I'm not running Nomad in a docker container; I'm running the native Docker client for mac. I think there's some missing communication though that's affecting both similarly.

[sethvargo]

Another interesting observation point: Nomad actually instructs the Docker client to download the image (and it does), but then it can't find the image inside the local docker registry to boot.

[joeshaw]

Another interesting observation point: Nomad actually instructs the Docker client to download the image (and it does), but then it can't find the image inside the local docker registry to boot.

@sethvargo, the "no such image" error is misleading. See fsouza/go-dockerclient#528

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.