Container image not found for docker job

[sronsiek]

Hi,

I'm running consul (0.7.0) + nomad client (0.4.1) + nomad server (0.4.1), each in separate docker containers on a cluster of 3 physical Linux hosts.

nomad server-members run in the server container looks good:

Name           Address        Port  Status  Leader  Protocol  Build  Datacenter  Region
swarm1.europe  192.168.100.1  4648  alive   true    2         0.4.1  dc1         europe
swarm2.europe  192.168.100.2  4648  alive   false   2         0.4.1  if_da       europe
swarm3.europe  192.168.100.3  4648  alive   false   2         0.4.1  if_da       europe

Running the following job:

job "first_job" {
  # Comment
  region = "europe"
  datacenters = ["dc1"]
  type = "service"

  group "common" {
    count = 1
    task "first_job_task" {
      driver = "docker"
      config {
        image = "endless_loop:1.0"
        network_mode = "host"
      }
      resources {
        cpu = 500 # 500 Mhz
        memory = 256 # 256MB
      }
    }
  }
}

... produces the following entries in the logs:

nomad-client1   |     2016/10/26 15:00:06.501089 [DEBUG] driver.docker: identified image endless_loop:1.0 as sha256:ce6937908b985320be76856be532793ee7ba65be02ddf177df23e8aca46ec948
nomad-client1   |     2016/10/26 15:00:06 [DEBUG] plugin: starting plugin: /usr/local/bin/nomad []string{"/usr/local/bin/nomad", "executor", "/tmp/client1/alloc/584d026a-02bd-07ed-b0a4-ac8d97e1a716/first_job_task/first_job_task-executor.out"}
nomad-client1   |     2016/10/26 15:00:06 [DEBUG] plugin: waiting for RPC address for: /usr/local/bin/nomad
nomad-client1   |     2016/10/26 15:00:06 [DEBUG] plugin: nomad: 2016/10/26 15:00:06 [DEBUG] plugin: plugin address: unix /tmp/plugin335853031
nomad-client1   |     2016/10/26 15:00:06.526392 [DEBUG] driver.docker: using 268435456 bytes memory for endless_loop:1.0
nomad-client1   |     2016/10/26 15:00:06.526453 [DEBUG] driver.docker: using 500 cpu shares for endless_loop:1.0
nomad-client1   |     2016/10/26 15:00:06.526486 [DEBUG] driver.docker: binding directories []string{"/tmp/client1/alloc/584d026a-02bd-07ed-b0a4-ac8d97e1a716/alloc:/alloc", "/tmp/client1/alloc/584d026a-02bd-07ed-b0a4-ac8d97e1a716/first_job_task:/local"} for endless_loop:1.0
nomad-client1   |     2016/10/26 15:00:06.526518 [DEBUG] driver.docker: allocated port 192.168.100.1:25159 -> 25159 (mapped)
nomad-client1   |     2016/10/26 15:00:06.526543 [DEBUG] driver.docker: exposed port 25159
nomad-client1   |     2016/10/26 15:00:06.526633 [DEBUG] driver.docker: setting container name to: first_job_task-584d026a-02bd-07ed-b0a4-ac8d97e1a716
nomad-client1   |     2016/10/26 15:00:06.528325 [ERR] driver.docker: failed to create container from image endless_loop:1.0: no such image

Here the first line says the docker image has been identified, that last line says no such image.

Seems contradictory - and the container is not launched.

[iverberk]

@sronsiek see this issue: #1080

Basically you can avoid a lot of problems by not running the Nomad client in a docker container. It is supposed to run and fully manage your node and it does not expect that it's own process is running in a sandboxed environment. That's where miscommunication with the docker daemon starts to happen because paths are being referred to that only exist in the Nomad client sandbox and not on the host, where the docker daemon is running.

[sronsiek]

I checked #1080, ksinica's comment is correct, the issue can be resolved by mounting the /tmp directory of the nomad client container on /tmp of the host.

@iverberk: Thanks for the pointer! Having committed to running applications in containers, it seems logical to do the same with nomad & consul, which provides common deployment and admin methods.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.