Permanent re-render template in job.

[sadpirit]

Nomad version

Output from nomad version
0.8.3

Operating system and Environment details

Ubuntu 16.04LTS

Issue

If the nomad template returns an dict, this results in a permanent restart

Reproduction steps

1. Place dict data in vault:

{
  "data": {
    "a": "b",
    "b": "b",
    "c": "b",
    "d": "b",
    "e": "b",
    "f": "b",
    "g": "b",
    "i": "b"
  }
}

2. make template in nomad job:

template {
       data = <<EOH
DATA_ACCESS_TOKEN={{with secret "secrets/ci/qa_"}}{{.Data.data}}{{end}}
EOH
       destination = "secrets/file.env"
       env = true
     }

3. Run job.

The job will restart indefinitely

Time                   Type              Description
05/28/18 08:51:52 STD  Started           Task started by client
05/28/18 08:51:52 STD  Restarting        Task restarting in 0s
05/28/18 08:51:52 STD  Killed            Task successfully killed
05/28/18 08:51:50 STD  Killing           Sent interrupt. Waiting 30s before force killing
05/28/18 08:51:50 STD  Restart Signaled  Template: template with change_mode restart re-rendered
05/28/18 08:45:56 STD  Started           Task started by client
05/28/18 08:45:56 STD  Restarting        Task restarting in 0s
05/28/18 08:45:56 STD  Killed            Task successfully killed
05/28/18 08:45:54 STD  Killing           Sent interrupt. Waiting 30s before force killing
05/28/18 08:45:54 STD  Restart Signaled  Template: template with change_mode restart re-rendered

If the data in the vaultare stored in an array, it will not result in an error. But vault returns the dictionary in random order.

bin# vault read secrets/ci/secrets/ci/qa_
data                map[f:b g:b i:b a:b b:b c:b d:b e:b]
bin# vault read secrets/ci/secrets/ci/qa_
data                map[b:b c:b d:b e:b f:b g:b i:b a:b]
bin# vault read secrets/ci/secrets/ci/qa_
data                map[a:b b:b c:b d:b e:b f:b g:b i:b]

[dadgar]

@sadpirit Can you file an issue against https://www.github.com/hashicorp/consul-template and link it to this.

[jippi]

thats a fun one, since a map in Go is not sorted, and will be random ordered on each creation - which is likely also why its re-rendering over and over :)

[dadgar]

@sadpirit I created the issue

[dansteen]

I seem to have this issue even when the data is not a map. As an example, given the following template stanza:

      template {
        data = <<EOH
# set the CONSUL_HTTP_TOKEN
CONSUL_HTTP_TOKEN="[[with secret (printf "consul/creds/traefik-%s" (env "CHEF_ENV"))]][[.Data.token]][[end]]"
EOH

        destination     = "secrets/file.env"
        left_delimiter  = "[["
        right_delimiter = "]]"
        env             = true
      }

I find that it is re-rendering the secrets/file.env constantly:

Recent Events:
Time                       Type              Description
2018-07-19T11:34:47-04:00  Started           Task started by client
2018-07-19T11:34:47-04:00  Restarting        Task restarting in 0s
2018-07-19T11:34:47-04:00  Killed            Task successfully killed
2018-07-19T11:34:42-04:00  Killing           Sent interrupt. Waiting 5s before force killing
2018-07-19T11:34:42-04:00  Restart Signaled  Template: template with change_mode restart re-rendered
2018-07-19T11:34:40-04:00  Started           Task started by client
2018-07-19T11:34:40-04:00  Restarting        Task restarting in 0s
2018-07-19T11:34:40-04:00  Killed            Task successfully killed
2018-07-19T11:34:35-04:00  Killing           Sent interrupt. Waiting 5s before force killing
2018-07-19T11:34:35-04:00  Restart Signaled  Template: template with change_mode restart re-rendered

As a note, the lease duration is set to quite high:

vault read consul/creds/traefik-stag                                                                                  
Key             Value
---             -----
lease_id        consul/creds/traefik-stag/47e349aa-7685-554f-aa3b-116859c1d4c6
lease_duration  720h0m0s
lease_renewable true
token           cad11817-6154-96cb-1855-980fc950a394

Nomad v0.8.3 (c85483d)
Debian Linux

[prologic]

I believe I'm running into this problem as well similar to @dansteen

My templates looks like this:

     template {
        data = <<EOH
          {
              "logLevel": "{{ env "LOG_LEVEL" }}",
              "mongoUri": "mongodb://{{ range $index, $service := service "mongod" }}{{ if ne $index 0 }},{{ end }}{{ $service.Address }}:{{ $service.Port }}{{ end }}/mydb?replicaSet=rs0",
              "xxx_host" : "{{ with node }}{{ .Node.Address }}{{ end }}",
              "xxx_port" : "1234",
              "xxx_admin_email": "{{ env "XXX_ADMIN_EMAIL" }}"
          }
        EOH

        destination = "myapp.cg"
      }

      template {
        data = <<EOH
          NODE_ENV = {{ key "/config/myapp/env" }}

          LOG_LEVEL = "{{ if keyOrDefault "/config/myapp/debug" "false" | parseBool }}debug{{ else }}info{{ end }}"

          XXX_ADMIN_EMAIL = "{{ key "/config/myapp/admin_email" }}"

          PORT = "{{ env "NOMAD_PORT_myapp" }}"

          HASH = "{{ key "/config/myapp/hash" }}"
          SALT = "{{ key "/config/myapp/salt" }}"

          CONFIG_FILE = "myapp.cfg"

          XXX_HOSTNAME = ""

          YYY_URI = "http://{{ with service "yyy-http-api" }}{{ with index . 0 }}{{ .Address }}:{{ .Port }}{{ end }}{{ end }}"
        EOH

        destination = "secrets/myapp.env"
        env         = true
      }

[prologic]

Actually my report/claim was false. But it does bring up an interesting question for me:

How do you debug Consul Template changes in Nomad? -- Or put another way; How do I poke around Logs, UI, something that tells me how/why a template has changed?

[prologic]

Actually my report/claim was false. But it does bring up an interesting question for me:

I had a dependent service that was in a restart loop with config that depended on it.

[Fuco1]

I'm also running into an infinite restart cycle and it's very hard to find out why. I'd also like to know more about the options we have to debug this, e.g. change logs of the template or diffs between re-renders.

[tgross]

I'm doing some old issue cleanup and it looks like this turned out to be a question rather than a bug, albeit one without much a satisfactory answer. If this comes up again for folks, let's open a new issue and make sure it gets properly triaged.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.