Nomad secrets backend ACL token ignores TTL and expires after 1 hour

[Thunderbottom]

Describe the bug
Nomad ACL tokens generated through the Vault Nomad Secrets Backend ignores set TTL (at nomad/config/lease) and expires after 1 hour of token generation.

To Reproduce
Steps to reproduce the behavior:

1. Enable the Vault Nomad secrets backend and add lease configuration:

$ vault write nomad/config/lease ttl=21600 max_ttl=21600

# or using the terraform vault provider:

resource "vault_nomad_secret_backend" "nomad" {
  backend     = "nomad"
  description = "Generate Nomad ACL tokens On-the-Fly"

  address = var.nomad_address
  token   = nomad_acl_token.vault_secrets_backend.secret_id

  # Valid for 6 hours by default
  default_lease_ttl_seconds = 21600
  max_lease_ttl_seconds     = 21600
  max_ttl                   = 21600
  ttl                       = 21600
}

2. Create a Nomad secrets backend role with type set to management, and read the token:

$ vault write nomad/role/role-name type=management
Success! Data written to: nomad/role/role-name

# or using the terraform vault provider

resource "vault_nomad_secret_role" "role_name" {
  role    = "role_name"
  backend = vault_nomad_secret_backend.nomad.backend
  type    = "management"
}

$ vault read nomad/creds/role-name

Key                Value
---                -----
lease_id           nomad/creds/role-name/1ju4torwBgCtKHvTG3Y1ndXh
lease_duration     6h
lease_renewable    true
accessor_id        1b486311-bb94-2743-1083-2c4d3b909131
secret_id          ec3a1769-c2fe-83f3-e69f-dbbf69deb6b9

Note the lease_duration (6h), which translates to the same lease duration in seconds (21600) set within the lease config of the Nomad secrets backend.

4. Set the token in the Nomad UI, and watch it expire after an hour.

Expected behavior
Nomad ACL tokens generated using the Nomad secrets backend should honor the lease duration (TTL) specified in nomad/config/lease

Environment:

• Vault Server Version (retrieve with vault status): 1.10.2
• Vault CLI Version (retrieve with vault version): 1.10.3
• Server Operating System/Architecture: Ubuntu 20.04, x86_64

Vault server configuration file(s):

$ cat /etc/vault.d/vault.hcl
ui = true
api_addr = "http://IP:8200"
cluster_addr = "http://IP:8201"

#mlock = true
#disable_mlock = true

storage "consul" {
  address = "consul.fqdn"
  path    = "vault"
  token   = "super-secret-token"
}

# HTTP listener
listener "tcp" {
  address = "IP:8200"
  cluster_address = "IP:8201"
  tls_disable = 1
}

seal "awskms" {
  region = "ap-south-1"
  kms_key_id = "<KMS-KEY>"
}

Additional context
Unsure whether this is an issue with Nomad, or with Vault. But in this case, this ACL token specifically happens to be generated by Vault Nomad secrets backend.

[Thunderbottom]

Anything on this yet? We thought this would be fixed with https://github.com/hashicorp/nomad/releases/tag/v1.3.3, specifically with hashicorp/nomad#13737, but it still is an issue.

Let me know if any more information is required, would be glad to comply.

[robmonte]

Hi @Thunderbottom
Sorry to hear you're having trouble with this. I tried to recreate this locally on my Mac using Vault version 1.10.5 but it seems to work as expected. Are you able to consistently recreate this issue? I can try again with your specific Vault version of 1.10.2.

In the meantime, could you try reading the lease information from Vault with the command:
vault lease lookup nomad/creds/role-name/1ju...

If you can perform this before the 1-hour mark and then do it again after 1-hour, we could see if your Vault is fully removing the lease after 1 hour or if something else may be going on.