raw_exec allows volume_mount stanza

[shantanugadgil]

Nomad version

Nomad v0.10.1 (0d4e5d9)

Operating system and Environment details

CentOS 7/8
Ubuntu 16.04/18.04

Issue

The raw_exec task driver allows the volume_mount stanza, even though it is not supported for raw_exec.

Reproduction steps

Use the job file specified; it should ideally fail the nomad plan step, but it doesn't.

Job file (if appropriate)

In the agent config:

client {
  enabled = true

  host_volume "repo_host" {
    path      = "/repo"
    read_only = true
  }

...

in the job definition:

group "work" {
    volume "repo_group" {
      source    = "repo_host"
      type      = "host"
      read_only = true
    }

    task "builder1" {
      driver = "raw_exec"

      volume_mount {
        volume      = "repo_group"
        destination = "/myrepo"
        read_only   = "true"
      }
...

Expected Behavior

The nomad plan step should flag/error that the "volume_mount" section is currently NOT supported for the raw_exec driver.

[tgross]

Hi @shantanugadgil and thanks for opening this issue! Yes, that looks like both a missing documentation item and a bug in the jobspec validation.

[shantanugadgil]

cross referencing #8262

[tgross]

Hey @shantanugadgil I wanted to circle back on this. The documentation fixes are in #8476 #8485 and you can see them on the task driver capabilities table for each driver.

Actually doing the validation ahead of time turns out to have some surprising challenges. Currently we don't push driver capabilities up to the server as part of the client fingerprint, so the server has no way of validating driver capabilities. We need the information from the client in order to account for capabilities changing between versions of a given driver, and with external drivers, the server might not even have the driver code at all to check!

That being said, getting this information up to the server doesn't seem like a fundamental design issue. I've worked up an issue to summarize the general problem and to get more design feedback on it: #9063

[shantanugadgil]

Thanks for taking the time to posting an update on this.

Updating documentation is fine, but in my opinion, the docs would help a lot those who might be starting out using Nomad.

For people who have been using Nomad quite a bit, and have been "pampered" with the "just works or tells you why it won't work", this was a bit confusing for me. 🥳

Initially, I had assumed that the mount would be a bind mount (like Docker mounts), which in all fairness is very much possible for raw_exec type drivers, right?

I would also like to cross reference this with #6627 just to keep it on the radar!

BTW, I am OK is this issue is closed!

[tgross]

For people who have been using Nomad quite a bit, and have been "pampered" with the "just works or tells you why it won't work", this was a bit confusing for me.

I'd definitely like this sort of thing to be more explicit, for sure. I've got some docs in the works to get into the details of the filesystem layout for tasks that I think can help here.

Initially, I had assumed that the mount would be a bind mount (like Docker mounts), which in all fairness is very much possible for raw_exec type drivers, right?

It's technically possible but because the raw_exec driver doesn't have a chroot for file system isolation, that lets would let the job create mounts anywhere on the host, which would effectively give raw_exec jobs root. We could probably come up with some sort of allow/denylist for that to restrict it to the task directory. But that's my understanding of the design decision there.

BTW, I am OK is this issue is closed!

Ok, closing in lieu of the larger problem in #9063. Thanks again for opening this!

[shantanugadgil]

I agree, once you are doing raw_exec it is pretty much "be careful about what you are doing" territory anyway, right?

@tgross any specific opinions about #6627

I already run the mount -t overlay overlay -o lowerdir=... commands via a shell script template block.

Having it more declarative would be much better (obvious), I think.

[tgross]

@tgross any specific opinions about #6627

It's marked for discussion in that issue.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.