Add support for -wait / quiescence in template{} stanza

[jippi]

Quiescence made a huge difference for us (bownty) when running CT, since our clusters are periodically very busy (5-10s) and then idle for hours and hours, saving us a lot of reloads of nginx

[lawliet89]

This issue is a little dated but I would like to chime in on this with my use case. I am currently using Vault to issue my jobs AWS credentials to write to S3. Since IAM is eventually consistent, there is a short period of time (probably < 10s) between Vault's creation of the IAM user and when the Nomad job is able to access AWS APIs with the credentials. Currently, our job is able to restart itself internally (without Nomad noticing) until the access succeeds. But it would be useful if we can configure the quiescence time.

I realise this is like #3866 wrt exposing more of Consul Template options. Probably requires some form of plugins interface that @preetapan mentioned.

[urjitbhatia]

This seems to be an old open issue but I am also running against this. We are trying to get membership information from Consul and when we do a big deployment, our listener service gets inundated with lots of reload signals in a very short time.

Any pointers on this?

[DhashS]

Quiescence solves a massive issue with determinism for me.
Currently, to expose the ZOOKEEPER_SERVERS environment variable to all (other) zookeeper containers, I have the following script.

{{ range $tag, $services := service "daemons-kafka-zookeeper|any" | byTag }}
{{ range $i, $services := service "daemons-kafka-zookeeper|any" }}
{{ end }}
{{ if $tag | contains "client" }}
{{ range $services }}
{{ scratch.MapSet .Address "client" .Port }}
{{ scratch.MapSet .Address "done" "false" }}
{{ end }}
{{ else }}
{{ if $tag | contains "follower" }}
{{ range $services }}
{{ scratch.MapSet .Address "follower" .Port }}
{{ scratch.MapSet .Address "done" "false" }}
{{ end }}
{{ else }}
{{ if $tag | contains "othersrvs" }}
{{ range $services }}
{{ scratch.MapSet .Address "othersrvs" .Port }}
{{ scratch.MapSet .Address "done" "false" }}
{{ end }}
{{ else }}
{{ end }}
{{ end }}
{{ end }}
{{ end }}
{{ scratch.Set "index" 1 }}
{{ scratch.Set "ZOO_SERVERS" "" }}
{{ range $i, $services := service "daemons-kafka-zookeeper|any" }}
{{ if ($services.Tags | contains "zookeeper-quorum") }}
{{ $addr := (scratch.Get .Address)}}
{{ if not (index $addr "done" | parseBool) }}
{{ $mynode := "temp" }}
{{ with node }}
{{ if eq $services.Node .Node.Node }}
{{ scratch.Set "ZOO_SERVERS" (print (scratch.Get "ZOO_SERVERS") "server." (scratch.Get "index") "=" $services.Address ":2888:3888;2181 " )}} <- this $services.Address may be 0.0.0.0
{{ scratch.Set "ZOO_MY_ID" (scratch.Get "index") }}
{{ else }}
{{ scratch.Set "ZOO_SERVERS" (print (scratch.Get "ZOO_SERVERS") "server." (scratch.Get "index") "=" $services.Address ":" (index $addr "follower") ":" ( index $addr "othersrvs") ";" (index $addr "client") " " )}}
{{ end }}
{{ scratch.MapSet $services.Address "done" "true"}}
{{ scratch.Set "index" (scratch.Get "index" | add 1 ) }}
{{ end }}
{{ end }}
{{ end }}
{{ end }}
ZOO_SERVERS="{{ or (scratch.Get "ZOO_SERVERS") (print "server." (env "NOMAD_ALLOC_INDEX") "=0.0.0.0:2888:3888;2181") }}"
ZOO_MY_ID={{or (scratch.Get "ZOO_MY_ID") (env "NOMAD_ALLOC_INDEX") }}
ZOO_PORT=2181

where the services can declare their ports via the tags on the service stanza in nomad.

However, the template thrashes when I launch it! It appears that there's no strict precedence order of templating and service registration. This can be solved by waiting for consensus among the template, or by giving a way for a template stanza to depend on another stanza starting first.

I saw some new task precedence stuff in the changelog for 0.9.0, so i'm going to be trying that out tonight

[cgbaker]

Just an update, this is on our roadmap for an upcoming release. We understand how useful this is.

[mikenomitch]

I believe this was closed out in 1.2.4 by this PR. Appreciate everybody's patience on this!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.