weird environment variables issue when run under process supervisor

[davidbirdsong]

OS: ubuntu 14.04
nomad v0.1.2

running under circus, circus job ini file:

[watcher:nomad-agent]
cmd = /bin/bash
args = /tmp/test.sh
numprocesses = 1
uid = root
working_dir = /tmp
priority = 10
singleton = True
stop_signal = SIGTERM
flapping.active = True
send_hup = True
stdout_stream.class = FileStream
stdout_stream.filename = /data/log/circusd/nomad-agent.stdout.log
stdout_stream.refresh_time = 0.3
stdout_stream.max_bytes = 209715200
stderr_stream.backup_count = 3
stderr_stream.class = FileStream
stderr_stream.filename = /data/log/circusd/nomad-agent.stderr.log
stderr_stream.max_bytes = 209715200
stderr_stream.refresh_time = 0.3
stdout_stream.backup_count = 3

/tmp/test.sh (illustrates PATH is set properly and uname is findable):

echo $PATH | tr ":" "\n"
echo -----
which uname
sleep 3

exec /tmp/nomad agent -client -servers 10.0.0.1:4646 -dc dc1 -network-interface em1 -data-dir /data/appdate/nomad-agent -meta public=0

stdout and stderr from circus:

==> /data/log/circusd/nomad-agent.stdout.log <==
PATH is 
/usr/local/bin
/usr/local/sbin
/usr/bin
/usr/sbin
/bin
/sbin
.
-----
/bin/uname


==> Starting Nomad agent...

==> /data/log/circusd/nomad-agent.stderr.log <==
==> Error starting agent: client setup failed: fingerprinting failed: Failed to run uname: exec: "uname": executable file not found in $PATH

to further verify that PATH is the problem, I checked out tag: v0.1.2 and made this change:

diff --git a/client/fingerprint/host.go b/client/fingerprint/host.go
index 5cbfee7..a616c84 100644
--- a/client/fingerprint/host.go
+++ b/client/fingerprint/host.go
@@ -3,6 +3,7 @@ package fingerprint
 import (
        "fmt"
        "log"
+       "os"
        "os/exec"
        "runtime"

@@ -36,6 +37,7 @@ func (f *HostFingerprint) Fingerprint(cfg *config.Config, node *structs.Node) (b
        node.Attributes["kernel.version"] = ""

        if runtime.GOOS != "windows" {
+               fmt.Printf("PATH is %s\n", os.Getenv("PATH"))
                out, err := exec.Command("uname", "-r").Output()
                if err != nil {
                        return false, fmt.Errorf("Failed to run uname: %s", err)

sure enough, running interactively PATH prints out, when run under the process supervisor, PATH is empty.

[cbednarski]

sure enough, running interactively PATH prints out, when run under the process supervisor, PATH is empty.

According to the circus docs, circus' environment is not copied to child processes by default. See the copy_env and copy_path options.

[davidbirdsong]

surprise, surprise. the shim bash script must have been a red herring--there's probably some implicit bash argument that simulates a login or something.

anyway, you're right.

[cbednarski]

👍 Thanks for following up!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.