Templates with vault database credentials don't re-render using a grace period

[deweller]

Nomad version

Nomad v0.5.6

Operating system and Environment details

Ubuntu Linux 16.04

Issue

I'm using a template to pull dynamic database credentials from Vault. When database credentials are rotated with Vault, there is a brief time while the app has stale database credentials. The credentials rotate in Vault as they should, but the app receives the updated credentials with no grace period. Therefore the app is working with stale credentials while the template re-renders.

Is possible to use the grace period in consul-template so the credentials are rotated a few minutes before the max_ttl expires on the database credentials?

Standard secrets reload at half of their lease, but the database credentials do not do this.

Reproduction steps

I created a new database role with a max_ttl of 1h. The database credentials are rotated and the template is re-rendered every hour and not every 30 minutes as expected.

My template data includes something like this:

{{ with secret "database/creds/myapp" }}
export DB_USERNAME="{{ .Data.username }}"
export DB_PASSWORD="{{ .Data.password }}"{{ end }}

Here is a sample of the nomad logs:

07/27/17 06:30:26 CDT  Signaling  Task being sent signal user defined signal 2: consul-template: template re-rendered
07/27/17 05:30:25 CDT  Signaling  Task being sent signal user defined signal 2: consul-template: template re-rendered
07/27/17 04:30:29 CDT  Signaling  Task being sent signal user defined signal 2: consul-template: template re-rendered

You can see that the template is re-rendered once per hour. I would expect the template to be re-render once every 30 minutes, or once every 55 minutes (with a 5 minute grace period).

[dadgar]

@deweller Is the ask to essentially expose the vault grace period in the config?

[deweller]

Yes. I'm requesting 2 things:

1. Verify that the grace setting applies to credentials like database/creds/myapp. I think it is possible that the default 15s (5m?) grace setting isn't working right now.
2. Expose the vault grace setting in the template stanza.

Thanks.

[dadgar]

@deweller Thanks for clarifying and for posting the issue! We are on the same page now.

Just so you are aware, the default grace is actually 5s: https://github.com/hashicorp/consul-template/blob/master/config/vault.go#L14

[deweller]

Great. Thanks.

BTW, I think that was intended to be 5 minutes.

hashicorp/consul-template@1d8e61f

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.