Vault client not active

[robottaway]

Nomad version

Nomad v0.5.0

Operating system and Environment details

Ubuntu 16.04 EC2 m3.xlarge

Issue

When I run a job that requires Vault integration token create fails:

Dec 27 21:45:23 stg-clustermgr-master08 nomad[20337]:     2016/12/27 21:45:23.287263 [ERR] nomad.node: Vault token creation failed: failed to create token for task "asky-log": Vault client not active
Dec 27 21:45:23 stg-clustermgr-master08 nomad[20337]:     2016/12/27 21:45:23.287663 [ERR] nomad.node: Vault token revocation failed: Vault client not active

Looking at the source code I think maybe the Vault client is only being set to active on the leader node? It appears that the job is being processed on stg-clustermgr-master08 which is not the leader node.

Reproduction steps

nomad run asky-log

watch logs with journalctl -f -n100 -u nomad.service on stg-clustermgr-master08

Job file (if appropriate)

job "asky-log" {
  type = "service"
  datacenters = ["us-west-2", "us-west-1"]

  group "asky-log" {
    count = 1

    vault {
      policies = ["asky-log"]
      change_mode   = "noop"
    }
      
    task "asky-log" {
      driver = "docker"

      config {
        image = "quay.io/rottaway/asky_log:newapitoken"
        port_map = {
          "http" = 5000
        }
        logging {
          type = "journald"
          config {
            tag = "${NOMAD_ALLOC_NAME}.${NOMAD_TASK_NAME}.${NOMAD_ALLOC_ID}"
          }
        }
      }

      service {
        tags = ["demo", "http", "rest"]
        name = "route-asky-log"
        port = "http"
        check {
          type = "http"
          protocol = "http"
          path = "/health"
          interval = "30s"
          timeout = "2s"
        }
      }

      resources {
        cpu = 200
        memory = 128
        network {
          mbits = 1
          port "http" {
          }
        }
      }
    }
  }
}

[dadgar]

Please upgrade to 0.5.2 and see if this is still a problem. PR #2050 likely fixed this and is merged in 0.5.1 and on.

[robottaway]

It seems to be working now but there is this odd message found in server logs when I run the job:

Dec 28 18:59:53 stg-clustermgr-master09 nomad[4474]:     2016/12/28 18:59:53.526733 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): <nil>

I can docker exec... into the asky-log container and find the token, then use the token to auth against vault and see that everything is in order. No idea on why the error related to DeriveVaultToken.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.