Data directory included in chroot causes infinite directory structure

[kyle-crumpton]

Nomad version

0.5.6

Operating system and Environment details

OS: Centos 7
Consul: 0.7.5

Issue

Job hangs on "Building task directory" when Nomad's data directory is included in the chroot (for example, using /etc/nomad.d/ as a data directory) until either the disk fills up, or memory runs out.

Reproduction steps

Set Nomad's data directory to /etc/nomad.d/, run a Job which utilizes a chroot.
Also, you can simply add your data directory to your "chroot_env" in the client config for Nomad. This will reproduce the same behavior.

Nomad Server logs (if appropriate)

Logs did not produce anything meaningful.

Nomad Client logs (if appropriate)

Setup Failure failed to build task directory for "example": Couldn't create symlink: symlink python2.7.1.gz /etc/nomad.d/alloc/ba334436-96e1-61bf-e6b1-9f8ff3c56a63/example/etc/nomad.d/alloc/ba334436-96e1-61bf-e6b1-9f8ff3c56a63/example/etc/nomad.d/alloc/2bb9b1a0-95c3-c06b-c7c9-6752eda18d2f/example/etc/nomad.d/alloc/2bb9b1a0-95c3-c06b-c7c9-6752eda18d2f/example/etc/nomad.d/alloc/b8318d40-4b90-5a65-90e5-99fdd57ec522/example/usr/share/man/man1/python2.1.gz: no space left on device

Job file (if appropriate)

job "sleep" {
     datacenters = ["us-central1-a-gce"]
     type = "service"
     group "sleep" {
       count = 1
       task "example" {
          driver = "exec"
          config {
               # When running a binary that exists on the host, the path must be absolute.
               command = "/bin/sleep"
               args    = ["1"]
           }
          resources {
	    memory = 500
          }
       }  
      }
}

A lot of this boils down to using a directory such as /etc to actively write data is typically a no-no. The issue is not limited to that, from what I've seen and reproduced, though. If I set the data directory in my chroot_env it will cause this issue regardless. It would be nice to detect this kind of thing and actively prevent the data directory and the chroot from overlapping, though. I didn't see any warnings in the server logs or anything about this in the documentation though.

[dadgar]

Yeah I agree we should do some amount of checking but this will largely come down to documentation and operator configuration.

[urjitbhatia]

We just got hit by this and it rendered our entire nomad worker cluster useless. This should be up in bold in the documentation.

[shantanugadgil]

I too experienced this recently. (Nomad v 0.8.4)

The Building task directory seems to be as if things are "hung" and not working.

The time taken it takes is not obvious to a new user and it takes a bit of "I have seen this symptom before" to understand what is going on rather than just by looking by the info/error messages.

Would there be a way too show some sort of progress indicator during the Building task directory bit?
(something analogous to the docker pulling layer progress indicator 😀 )

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.