Workload Identity

[mikenomitch]

Proposal

As part of the Variables project for Nomad 1.4, each Nomad Task is given a signed token that encodes information about its namespace, job, task group, and task. This Nomad Workload Identity JWT is used in order to validate that the Nomad task in question has access to the secure variables requested.

We should expand upon this work in several ways:

• Task should have access to their Workload Identity token and Nomad’s public key
• Add Standard OIDC keys and values to workload identity tokens
• A JWKS URL for public keys should be added
• An OIDC Configuration URL for OIDC Discovery should be added

This would allow us to greatly expand the use of the workload identity tokens for more use cases.

Use-cases

• This would allow users to use Nomad as an OIDC provider. They could configure OIDC consumers to give access to resources based on identity. For instance, AWS resources such as S3 buckets could be exposed to certain jobs. Azure and GCP have similar mechanisms
• This would allow users to have fine grained permission checks based on JWT. For instance, if a certain route in app A should only allow requests from app B, the JWT from app B could prove identity and grant permission.
• This could be used as the basis to redo the Vault integration to remove the need for Vault ACL Tokens on every job, allow for multiple Vault clusters per Nomad cluster, and make it easier to run Vault as a Nomad job
• This could be used as the basis to redo the Consul integration to remove the need for Consul ACL Tokens on every job, allow for multiple Consul clusters per Nomad cluster, and make it easier to run Consul as a Nomad job

[anarsen]

Absolutely love that this is coming to the Hashi stack.

You mention using OIDC federation, allowing Nomad tasks to assume an AWS IAM role using the task local JWT token issued by Nomad as an example.

For this to work, the OIDC endpoints in Nomad would have to be exposed on the public Internet. While there's nothing inherently sensitive in these endpoints, HashiCorp have always recommended not exposing Nomad endpoints publicly, nor Consul or Vault for that matter. Regardless of ACL, TLS etc. At least that's been my impression.

Would your docs on this end up recommending something like exposing only the OIDC endpoints publicly then?

[mikenomitch]

@anarsen, the public URL issue is something we'll still need to figure out.

To be honest, I'm not sure what our solution will be, but I'm glad you've called it out here, as it'll force us to think about this early. I wouldn't want to force people to deploy a job to proxy only specific routes to the Nomad servers, but that'd be at least one way to expose just these routes. If there's some other option you, or anybody else reading this, has in mind, let me know.

I'll update this thread once we zero in on a solution for this.

[anarsen]

My initial thought is that the OIDC configuration endpoints are cacheable, and offloading the Nomad servers from serving these sounds like a good idea.

Proposal 1: OIDC agent

A separate Nomad agent serving only OIDC configurations. Nomad servers would push OIDC config to these agents when there are updates.

Scalable and isolated from the Nomad server memory space. Minimizes the risk of someone exposing sensitive Nomad Server endpoints publicly. However, it's yet another thing to deploy and operate.

Proposal n: TBD

[tgross]

Related PRs slated for the 1.5.0 release:

• WI: allow workloads to use RPCs associated with HTTP API #15870
• Task API via Unix Domain Socket #15864
• Add option to expose workload token to task #15755

[mcpherrinm]

The OIDC /.well-known/openid-configuration and JWKS keys are relatively static I believe, and can be uploaded to a public, static webhost. AWS' own OIDC documentation suggests using S3 for your JWKS. Perhaps Nomad could include support for storing those in an S3 bucket or compatible object storage, or include a new plugin type to handle publicly exposing those files.

Even if this isn't built into Nomad, it would be easy to have an external utility which scrapes and uploads them.

[tgross]

Shipped in Nomad 1.7.0-beta.1