using exec driver result into RPC error

[manish-panwar]

Nomad version

v0.9.1

Operating system and Environment details

Photon OS

Issue

I am trying to create a directory using exec driver and it result into cryptic error message.

Snippet from Nomad Job:

task "zk-config-dir" {
driver = "exec"
config {
command = "mkdir"
args = [ "-p", "/opt/zookeeper/config" ]
}
}

Please make a note that I tried providing the absolute path for mkdir but don't work either. Right now I am running everything as root to rule out any permission issue.

Nomad Client logs (if appropriate)

2019-07-19T17:37:10.466Z [INFO ] client.driver_mgr.exec: starting task: driver=exec driver_cfg="{Command:/bin/mkdir Args:[-p /opt/zookeeper/manish]}"
2019-07-19T17:37:10.466Z [DEBUG] client.driver_mgr.exec.executor: starting plugin: alloc_id=c0d0361a-e895-0bda-a193-9f365535040a driver=exec task_name=zk-config-dir path=/usr/local/bin/nomad args="[/usr/local/bin/nomad executor {"LogFile":"/opt/nomad/alloc/c0d0361a-e895-0bda-a193-9f365535040a/zk-config-dir/executor.out","LogLevel":"debug","FSIsolation":true}]"
2019-07-19T17:37:10.467Z [DEBUG] client.driver_mgr.exec.executor: plugin started: alloc_id=c0d0361a-e895-0bda-a193-9f365535040a driver=exec task_name=zk-config-dir path=/usr/local/bin/nomad pid=1356
2019-07-19T17:37:10.467Z [DEBUG] client.driver_mgr.exec.executor: waiting for RPC address: alloc_id=c0d0361a-e895-0bda-a193-9f365535040a driver=exec task_name=zk-config-dir path=/usr/local/bin/nomad
2019-07-19T17:37:10.486Z [DEBUG] client.driver_mgr.exec.executor: using plugin: alloc_id=c0d0361a-e895-0bda-a193-9f365535040a driver=exec task_name=zk-config-dir version=2
2019-07-19T17:37:10.550Z [ERROR] client.driver_mgr.exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=c0d0361a-e895-0bda-a193-9f365535040a driver=exec task_name=zk-config-dir error="rpc error: code = Unavailable desc = transport is closing"
2019-07-19T17:37:10.550Z [DEBUG] client.alloc_runner.task_runner.task_hook.stats_hook: error fetching stats of task: alloc_id=c0d0361a-e895-0bda-a193-9f365535040a task=zk-config-dir error="rpc error: code = Canceled desc = grpc: the client connection is closing"

Logs from executor.out

{"@Level":"debug","@message":"plugin address","@timestamp":"2019-07-19T18:44:53.594888Z","address":"/tmp/plugin754277908","network":"unix"}
{"@Level":"debug","@message":"launching command","@module":"isolated_executor","@timestamp":"2019-07-19T18:44:53.596204Z","args":"-p /opt/zookeeper/config","command":"mkdir"}

[notnoop]

Mind if you provide result of nomad alloc status <alloc-id> as well?

The "errors" in the log files are red herring and expected when task completed. The alloc status command should who the actual status of command and whether we failed it.

There are potentially few issues that masks the issue here:

• As you noted, mkdir -p /opt/zookeeper/config may require running root as nobody, the default user may not have write access to /opt or /.
• With exec driver, the command runs in an ephemeral chroot environment. So the directory created in the chroot environment will not be visible on the host path or other tasks after the task completes.
• If the task job is not a batch one, the task exiting is interpreted as a failure and nomad will restart the task.

[stale]

This issue will be auto-closed because there hasn't been any activity for a few months. Feel free to open a new one if you still experience this problem 👍

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.