Vault integration

[c4milo]

Are there any plans to integrate Vault with Nomad? It seems it would be very useful to be able to query secrets and pass them as ENV variables to the different tasks.

[cbednarski]

Thanks for opening an issue. This is something we're thinking about.

"Vault integration" is a large and complex topic. Rather than considering how Vault will integrate with Nomad as a whole, it's easier to think how we can use Vault to implement specific features in Nomad. For example, we want to provide secure communication between agents, provide authentication and authorization for agents and users, store secrets, etc. Each of these use cases needs a different design and implementation, and Vault may help us implement these features.

If you can share your use cases that will help us design and prioritize those particular features.

[pires]

I want to be able to deliver the following to my apps:

• Tokens, i.e. OAuth2 or cloud provider credentials
• Credentials for using with external services, i.e. databases
• SSL certificates for TLS
• Encryption keys for using encrypted persistent volumes (hint at possible future support to persistent volumes other than the host's, i.e. EBS volume or GCE Persistent Disk.

[diptanu]

@pires It would be nice to understand the UX around each of these and how they relate to Nomad?

For eample, what is stopping an application running on Nomad to not directly use Vault to get credentials of an external service? Or if the Nomad client gets it, how does it hands it over to the process running as a Nomad Task?

[c4milo]

@diptanu, there is less friction in adoption if you just assume people are not going to be using directly Vault or Consul, for instance. So, providing mechanisms to leverage those technologies outside of the applications themselves is a good way of speeding up adoption. A good example is consul-template.

However, for people that like to make configurations part of the application container, packaging it as part of a release, consul-template is not very helpful unless they are willing to load their configurations from external volumes. A much better scenario in terms of portability and reliability, in my opinion, is being able to pass Vault values from Nomad into tasks, through environment variables.

The implementation could be something similar to what Terraform does with the Interpolation syntax, supporting a vault(key) function that invokes Vault to read values only.

[c4milo]

I'm thinking in something like this:

task "registry" {
  driver = "docker"
  env {
    LOG_LEVEL="debug"
    REGISTRY_STORAGE="s3"
    REGISTRY_STORAGE_S3_ACCESSKEY="${vault("/aws/keys/access")}"
    REGISTRY_STORAGE_S3_SECRETKEY="${vault("/aws/keys/secret")}"
    REGISTRY_STORAGE_S3_REGION="us-east-1"
    REGISTRY_STORAGE_S3_BUCKET="mybucket"
    REGISTRY_STORAGE_S3_ROOTDIRECTORY="/foo/bar/docker-registry"
  }
  config {
    image = "registry:2"
    network_mode = "host"
  }
  resources {
    cpu = 500 # Mhz
    memory = 128 # MB
    network {
      mbits = 10
      port "http" {}
    }
}

[pires]

Exactly what @c4milo is proposing!

[ketzacoatl]

yes, +1 to that!

[adrianlop]

+1 too, we're gonna need it for production.

[trong]

REGISTRY_STORAGE_S3_ACCESSKEY="${vault("/aws/keys/access")}"

It's great!!!

[BSick7]

+1

If this secret was granted on a lease, would nomad restart the container when the credentials expired?

[jhartman86]

+1 for @c4milo's use case!

[dvusboy]

While @c4milo's proposed syntax is easy and convenient, it is also incredibly insecure. A simple docker inspect --format='{{ .Config.Env }}' <id> will dump all those secrets in the clear. It is marginally better than just providing the secrets in plain-text because you can check the vault syntax into revision control.

[c4milo]

True, the example is intended to show how the built-in function could be used. Passing secrets through environment variables is not safe if a person different to you has access to your host.

[dvusboy]

It wasn't clear to me that's obvious, judging from all the +1. In fact, in the specific case of the Docker task driver, it is a very tricky thing to accomplish. You would almost need a specialized container entry-point to 'unwrap' the secrets from the encrypted environment variables.

[pires]

While I agree with @dvusboy concern - and @c4milo justification -, I wonder if someone owns your Nomad machines, which is needed for you to run such docker commands, how would not using env vars to expose secrets help you in any way to protect your system.

[troyswanson]

I'm a novice in all of this, so it's very possible that I'm off base. Please correct me if I am wrong.

However, is it too much to ask for the job to be responsible for asking Vault for the secrets itself? For instance, if Vault is being hosted inside a cluster alongside Nomad, I imagine it should be possible for the application to have logic to obtain the secrets that Vault contains.

[zapman449]

If someone can interact with the docker API / socket, your entire box and all it's data are owned. Nothing can be done about it.

The biggest use case of this is for container orchestration. Moving a container from Dev to Qa to Prod you want different credentials everywhere, and baking them into the application is not prudent at all.

If you offload the problem to a hypothetical Nomad+vault integration, the orchestration becomes much more powerful. Even better, the same job specification can be used in dev/qa/prod rather than evolving by re-writing the credentials per environment.

The point about pushing integration down into the app itself is a far higher burden, as that effort needs to be duplicated in any and all apps which are deployed.

[jgeiger]

I've been thinking about this for a while, and my best case use would be something like:

1. Scheduler creates container ID.
2. Scheduler uses ID as a one time use token in vault for a specific set of configuration credentials.
3. Scheduler launches container passing ID as an ENV variable.
4. App in container launches using passed in ID to get config from vault holding the info in memory.

With this setup nothing is ever saved on disk or in the container since the passed ENV var is one time use and the app is still able to configure itself via vault. You still need to have your application call vault on boot, but it's really no different than any other method. You could also use something like a Config Interface, where you could create different versions depending on where you were running (VaultConfg, EnvVarConfig, FileConfig, etc)

Just a note, I'm not a current user of Nomad, but this workflow is what I'll need in a scheduler to use it for production and I think it's pretty doable.

[steve-jansen]

I like your thinking @jgeiger. One challenge is supporting Vault's rich concept for leases and dynamic secrets.

Expanding on your suggestion:

1. Optional list of Vault policies in the job spec
2. Optional Vault token in job spec, or use conf block for Vault access
3. Nomad creates a new token with desired policies using the supplied token/conf
4. Nomad saves the token to a cubbyhole
5. For each container instance, Nomad creates a one-time-use token to read the cubbyhole from that single container
6. Nomad injects the one-time-use token as the env vars VAULT_ADDR, VAULT_CUBBYHOLE_TOKEN, and VAULT_CUBBYHOLE_PATH.

The one-time-use token exposed in the env (e.g. docker inspect) becomes useless after a short TTL, or first use of the cubbyhole token.

[F21]

This would be really cool with the token wrapping functionality that was introduced in Vault 0.6.0: https://www.hashicorp.com/blog/vault-0.6.html

With token wrapping, nomad won't be able to see the real token and will not need to manage the cubbyhole itself:

1. Nomad creates the token using the API equivalent of vault token-create -policy=default -wrap-ttl=60s
2. It passes the wrapping_token, ttl, and creation_time to the job, so that the job can use the ttl and creation_time to check if someone else used the token and raise a security alert.
3. The job asks vault to unwrap the token and gets the real token back.

[bryantidd]

I like @F21 's notion, only concern is what happens if/when you need to restart an existing container, would that require modifying the job def and update when you just need to bounce a running container?
I think the process to get good secrets injected into a container on initial start is complex, but not complicated, based on much of the understanding discussed here. The next interesting issue is in supporting re-injection on restart, upgrade, etc??

[dadgar]

Hey this is coming in 0.5 and a lot of it is in master already

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.