Unable to retrieve template secrets from Vault

[aedades]

Nomad version

Nomad v0.8.4 (dbee1d7)

Operating system and Environment details

Linux ip-10-6-18-143 4.14.51-60.38.amzn1.x86_64 #1 SMP Tue Jun 26 23:06:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Issue

Unable to retrieve secrets from Vault (version 0.11.1), though I believe Nomad servers and clients have been configured properly:

# Server:
  "vault": {
    "enabled": true,
    "address": "<vault address>",
    "token": "<root token>"
  },

# Client:
  "vault": {
    "enabled": true,
    "address": "<vault address>"
  },

Reproduction steps

Try to deploy a job with the below template and the above Nomad configuration.

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

    2018/09/25 22:54:29.370930 [WARN] (view) vault.read(secret/.../password): vault.read(secret/.../password): Error making API request.
URL: GET <vault address>/v1/secret/.../password
    2018/09/25 22:54:29.370998 [WARN] (view) vault.read(secret/.../api_key): vault.read(secret/.../api_key): Error making API request.
URL: GET <vault address>/v1/secret/.../api_key
    2018/09/25 22:55:29.440031 [ERR] (view) vault.read(secret/.../password): vault.read(secret/.../password): Error making API request.
URL: GET <vault address>/v1/secret/.../password
    2018/09/25 22:55:29.440094 [ERR] (runner) watcher reported error: vault.read(secret/.../password): vault.read(secret/.../password): Error making API request.
...
* missing client token (retry attempt 1 after "250ms")

Job file (if appropriate)

...
          "Templates": [
            {
              "SourcePath": "",
              "DestPath": "secrets/rendered.env",
              "EmbeddedTmpl": "PASSWORD=\"{{with secret \"secret/.../password\"}}{{.Data.value}}{{end}}\nAPI_KEY=\"{{with secret \"secret/.../api_key\"}}{{.Data.value}}{{end}}\"\n",
              "ChangeMode": "restart",
              "ChangeSignal": "",
              "Splay": 5000000000,
              "Perms": "0644",
              "LeftDelim": "{{",
              "RightDelim": "}}",
              "Envvars": true,
              "VaultGrace": 15000000000
            }
          ],
...

[schmichael]

There is a known bug with Nomad v0.8.4 and Vault 0.11.x: #4633 (PR: #4658)

Would you be able to test with Nomad v0.8.5?

[dadgar]

@aedades Have you included a vault stanza in the task. Otherwise you are trying to request secrets without credentials

[aedades]

Thanks @schmichael , @dadgar , I'm still seeing the same error, but I've updated to nomad 0.8.5, and have added

    "Vault": {
      "policies": ["<policy name>"]
    }

to the job spec (at the job level).

[aedades]

I was able to get this working after moving the vault stanza to the task level. It looks like the missing vault stanza was the main culprit - the job successfully deploys with this spec in both 0.8.5 and 0.8.4 nomad environments. Thank you both!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.