templates environment variables aren't interpreted consistently

[notnoop]

Nomad version

Nomad v0.8.7 (21a2d93)
Also tested against Nomad master (ce95d5e).

Operating system and Environment details

macOS and running agent in dev mode

Issue

In a task template stanza, when the destination field contains an environment variable reference, launching the task fails if env=true, but succeeds and works as expected otherwise.

Documentation hints that environment variable references are allowed in environment fields, e.g. [1], but it's not consistently applied.

[1] https://www.nomadproject.io/docs/job-specification/template.html#pki-certificate

Reproduction steps

For testing the successful case when env=false,

1. Submit a job with job file below
2. inspect the resulting job and alloc status, and check the presence of /secrets/file.env in the resulting container

Then uncomment the # env = true line, and submit the job again: expect job to run, but will find Template: error opening env template: open /private/var/folders/.../${NOMAD_SECRETS_DIR}/file.env: no such file or directory error

My reproduction steps and output are:

$ nomad --version
Nomad v0.8.7 (21a2d93eecf018ad2209a5eab6aae6c359267933)
$ # successful case
$ nomad job run ./example-ct-envvar.nomad
==> Monitoring evaluation "18e00c1f"
    Evaluation triggered by job "example-ct-envvar"
    Allocation "3ebae838" created: node "3843bc43", group "cache"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "18e00c1f" finished with status "complete"
$ # check file presence for the container of the alloc
$ docker exec $(docker ps |grep 3ebae838 | awk '{print $1;}' ) cat /secrets/file.env
FOO=BAR



$ # Now let's test with env=true
$ cat example-ct-envvar.nomad | sed 's|# *env|env|g' > example-ct-envvar-set.nomad
$ nomad job run ./example-ct-envvar-set.nomad
==> Monitoring evaluation "d86e2411"
    Evaluation triggered by job "example-ct-envvar"
    Allocation "d4e8a036" created: node "3843bc43", group "cache"
    Allocation "d4e8a036" status changed: "pending" -> "failed"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "d86e2411" finished with status "complete"
$ nomad alloc status d4e8a036
ID                     = d4e8a036
Eval ID                = d86e2411
Name                   = example-ct-envvar.cache[0]
Node ID                = 3843bc43
Job ID                 = example-ct-envvar
Job Version            = 1
Client Status          = failed
Client Description     = <none>
Desired Status         = run
Desired Description    = <none>
Created                = 11s ago
Modified               = 10s ago
Reschedule Eligibility = 20s from now

Task "redis" is "dead"
Task Resources
CPU      Memory   Disk     IOPS  Addresses
500 MHz  256 MiB  300 MiB  0

Task Events:
Started At     = N/A
Finished At    = 2019-01-28T13:51:44Z
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type        Description
2019-01-28T08:51:44-05:00  Killing     Template: error opening env template: open /private/var/folders/r6/346cfqyn76b_lx1nrcl5278c0000gp/T/NomadClient044727626/d4e8a036-77bf-68fa-8f0f-7771cb29f285/redis/${NOMAD_SECRETS_DIR}/file.env: no such file or directory
2019-01-28T08:51:44-05:00  Task Setup  Building Task Directory
2019-01-28T08:51:44-05:00  Received    Task received by client

Job file (if appropriate)

job "example-ct-envvar" {
  datacenters = ["dc1"]


  group "cache" {
    task "redis" {
      driver = "docker"

      config {
        image = "redis:3.2"
      }

      resources {
        cpu    = 500
        memory = 256
      }

      template {
        data = <<EOF
FOO=BAR
EOF
        destination   = "${NOMAD_SECRETS_DIR}/file.env"
	# env = true
      }

    }
  }
}

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.