Template stanza errors creating credentials from vault nomad backend.

[adragoset]

Nomad version

Nomad v0.7.1 (0b295d3)
Vault v0.9.3
Consul v1.0.6

Operating system and Environment details

Alpine 3.7.0

Issue

When using the template stanza consul-template fails to create nomad token from the vault nomad backend.

I have a job with the relevant template section as follows.

 {
                          {
                                {% raw %}
                                "EmbeddedTmpl": 'CONSUL_ACL_TOKEN="{{with secret "/consul/creds/hashiui_consul_mgt"}}{{.Data.token}}{{end}}"',
                                {% endraw %}
                                "DestPath": "local/consul_acl.env",
                                "Envvars": true,
                                "ChangeMode": "restart"
                            },
                            {
                                {% raw %}
                                "EmbeddedTmpl": 'NOMAD_ACL_TOKEN="{{with secret "/nomad/creds/mgt"}}{{.Data.secret_id}}{{end}}"',
                                {% endraw %}
                                "DestPath": "local/nomad_acl.env",
                                "Envvars": true,
                                "ChangeMode": "restart"
                            }
}

I have the nomad and consul backend mounted in vault

 vault mounts list
Path        Type       Accessor            Plugin  Default TTL  Max TTL  Force No Cache  Replication Behavior  Seal Wrap  Description
consul/     consul     consul_1d02d76b     n/a     system       system   false           replicated            false    Consul acl tokens
cubbyhole/  cubbyhole  cubbyhole_8ad0619d  n/a     n/a          n/a      false           local                 false      per-token private secret storage
database/   database   database_e17f02d8   n/a     8760         8760     false           replicated            false      Generate database credentials
identity/   identity   identity_7e78ccc8   n/a     n/a          n/a      false           replicated            false      identity store
nomad/      nomad      nomad_74ca4526      n/a     system       system   false           replicated            false      Nomad acl tokens
pki/        pki        pki_7278c4f6        n/a     8760         8760     false           replicated            false      Generate pki certificates
secret/     kv         kv_55bf8d26         n/a     system       system   false           replicated            false      key/value secret storage
sys/        system     system_bd9db6ac     n/a     n/a          n/a      false           replicated            false      system endpoints used for control, policy and debugging
transit/    transit    transit_eeed77d8    n/a     system       system   false           replicated            false

The consul/role/hashiui_consul_mgt config

curl -k --header "X-Vault-Token: <my root token>" https://vault.service.consul:8200/v1/consul/roles/hashiui_consul_mgt --cert nomad-cli.pem --key nomad-cli-key.pem
{"request_id":"65e250f7-850d-f005-8ea4-13b52b8b067d","lease_id":"","renewable":false,"lease_duration":0,"data":{"lease":0,"token_type":"management"},"wrap_info":null,"warnings":null,"auth":null}

The nomad/role/mgt config

curl -k --header "X-Vault-Token: <my root token>" https://vault.se
rvice.consul:8200/v1/nomad/role/mgt --cert nomad-cli.pem --key nomad-cli-key.pem
{"request_id":"7ad2a0bd-847c-6e36-30e2-3c6f21a41d18","lease_id":"","renewable":false,"lease_duration":0,"data":{"global":false,"policies":[],"type":"management"},"wrap_info":null,"warnings":null,"auth":null}

I can read creds from the nomad and consul backend as follows with vaults root token

 vault read consul/creds/hashiui_consul_mgt
Key             Value
---             -----
lease_id        consul/creds/hashiui_consul_mgt/6cb06cc7-bfa0-9988-3d43-6814d5632b0f
lease_duration  168h0m0s
lease_renewable true
token           e235dd35-e9fb-f5b1-3739-4b3db3dc3232

vault read nomad/creds/mgt
Key             Value
---             -----
lease_id        nomad/creds/mgt/5d4d2947-48d6-1530-faeb-1703cf5eea26
lease_duration  168h0m0s
lease_renewable true
accessor_id     8c1d7826-4424-31b2-7fdb-e88bdaa9a18f
secret_id       d731f89c-8743-cb60-5ec4-e9cfed3e5b72

Nomad is configured with vaults root token for vault access.

The consul token reads correctly if i remove the template for the nomad/creds/mgt token the job succeeds.

The following is outputted by allocation status indicating job failure trying to create the nomad token.

ID                  = eec197b4
Eval ID             = f1e2be60
Name                = hashiui.web[0]
Node ID             = 4b374149
Job ID              = hashiui
Job Version         = 7
Client Status       = failed
Client Description  = <none>
Desired Status      = run
Desired Description = <none>
Created             = 1h4m ago
Modified            = 59m31s ago
Deployment ID       = e627a75f
Deployment Health   = unhealthy

Task "frontend" is "dead"
Task Resources
CPU      Memory   Disk     IOPS  Addresses
250 MHz  128 MiB  300 MiB  0     http: 192.168.32.147:22303

Task Events:
Started At     = N/A
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                   Type             Description
02/16/18 16:16:46 UTC  Killing          Killing task: Template: vault.read(nomad/creds/mgt): vault.read(nomad/creds/mgt): Error making API request.

URL: GET https://vault.service.consul:8200/v1/nomad/creds/mgt
Code: 500. Errors:

* 1 error occurred:

* Unexpected response code: 500 (3 error(s) occurred:

* RPC failed to server 192.168.32.144:4647: rpc error: token 0 invalid: 1 error(s) occurred:

* token name too long
* RPC failed to server 192.168.32.143:4647: rpc error: rpc error: token 0 invalid: 1 error(s) occurred:

* token name too long
* RPC failed to server 192.168.32.145:4647: rpc error: rpc error: token 0 invalid: 1 error(s) occurred:

* token name too long)
02/16/18 16:16:41 UTC  Alloc Unhealthy  Task not running by deadline
02/16/18 16:11:45 UTC  Template         Missing: vault.read(nomad/creds/mgt)
02/16/18 16:11:41 UTC  Task Setup       Building Task Directory
02/16/18 16:11:41 UTC  Received         Task received by client

Was considering this may be a consul template issue if that proves to be the case ill open a ticket in consul templates repository.

I'm actually leaning towards this being either a failure in consul templates vault api version or a failure in the vault api itself. It looks like nomad is using the latest release version of consul template however there are a few newer versions of the vault api since consul template was last released.

[preetapan]

Related PR on vault that trims ACL token names - hashicorp/vault#4009

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.