Allocation fails with mount: operation not permitted

[kaskavalci]

Nomad version

v0.5.1-rc1

Operating system and Environment details

golang:1.7.3 docker image
nomad is running in dev mode with su permissions
root 30 24 0 08:58 ? 00:00:17 nomad agent -dev

Issue

Same job file works in v0.4.0. After 0.5.0 upgrade (and rc1 as well) nomad fails to run the job.

Reproduction steps

Run the following job in golang docker container.

PS: Job tested with non-docker ubuntu host does not have that problem.

Nomad Server logs (if appropriate)

    2016/12/08 12:37:02.494021 [DEBUG] worker: dequeued evaluation 4ce5e34d-7b96-40c0-7fe0-feaf93f586c1
    2016/12/08 12:37:02.494126 [DEBUG] sched: <Eval '4ce5e34d-7b96-40c0-7fe0-feaf93f586c1' JobID: 'test'>: allocs: (place 1) (update 0) (migrate 0) (stop 0) (ignore 0) (lost 0)
    2016/12/08 12:37:02.494674 [DEBUG] worker: submitted plan for evaluation 4ce5e34d-7b96-40c0-7fe0-feaf93f586c1
    2016/12/08 12:37:02.494694 [DEBUG] sched: <Eval '4ce5e34d-7b96-40c0-7fe0-feaf93f586c1' JobID: 'test'>: setting status to complete
    2016/12/08 12:37:02.494820 [DEBUG] worker: updated evaluation <Eval '4ce5e34d-7b96-40c0-7fe0-feaf93f586c1' JobID: 'test'>
    2016/12/08 12:37:02.494863 [DEBUG] worker: ack for evaluation 4ce5e34d-7b96-40c0-7fe0-feaf93f586c1
    2016/12/08 12:37:02.494920 [DEBUG] client: updated allocations at index 8 (pulled 1) (filtered 0)
    2016/12/08 12:37:02.494941 [DEBUG] http: Request /v1/jobs?region=global (1.949825ms)
    2016/12/08 12:37:02.494972 [DEBUG] client: allocs: (added 1) (removed 0) (updated 0) (ignore 0)
    2016/12/08 12:37:02.495927 [DEBUG] http: Request /v1/evaluation/4ce5e34d-7b96-40c0-7fe0-feaf93f586c1?region=global (140.494µs)
    2016/12/08 12:37:02.495952 [WARN] client: failed to build task directories: mount: operation not permitted
    2016/12/08 12:37:02.496677 [DEBUG] http: Request /v1/evaluation/4ce5e34d-7b96-40c0-7fe0-feaf93f586c1/allocations?region=global (160.878µs)
    2016/12/08 12:37:02.626582 [DEBUG] client: updated allocations at index 10 (pulled 0) (filtered 1)
    2016/12/08 12:37:02.626915 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)

Job file (if appropriate)

job "test" {
    datacenters = ["dc1"]

    type = "service"

    group "test" {
        count = 1
        restart {
            interval = "5m"
            attempts = 20
            delay = "10s"
            mode = "delay"
        }
        task "test" {
            driver = "raw_exec"
            config {
                command= "/bin/sleep"
                args = ["1000"]
            }

            #Discovery Service Registration
            service {
                name = "test"
                tags = ["test-tag"]
                check {
                  type     = "script"
                  name     = "check_table"
                  command  = "/bin/sh"
                  args     = ["-c", "whoami"]
                  interval = "60s"
                  timeout  = "5s"
                }
            }

            resources {
                cpu = 100
                memory = 100
                network {
                    mbits = 1
                }
            }
        }
    }
}

[dadgar]

@kaskavalci Just need to run the container with --priviledged. moby/moby#5254

Confirmed it worked! If it works for you lets close out the issue! Thanks

[kaskavalci]

Can confirm priviledged mode solves this issue. 👍

[matthanley]

Does anyone know of a workaround to this issue without --privileged? I'm deploying Nomad clients via containers through Docker Swarm so privileged mode isn't an option.

[kaskavalci]

Hi @matthaias I think it is on the way #2178

[shantanugadgil]

@dadgar @matthanley
I hit this issue while starting up docker containers which were executing "ulimit -n <large_number>" in the entrypoint script.
Setting privileged to true solved the issue for me.

Do you think this could be mentioned in the docker driver docs with the word "ulimit" in there somewhere? Could possibly ease people searching for "nomad + docker + ulimit" ?

Regards,
Shantanu

[dposton80]

I found that if you don't specify a USER in your docker container, nomad can fail with a similar error (even though privileged mode is no longer officially required):
See #10049

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.