JWT validation fails after EKS OIDC provider's key rotation

[natefive]

We're using EKS service account tokens to authenticate via OAuth2 proxy. When the EKS OIDC provider rotates it's keys the OAuth2 proxy/go-oidc client does not update it's key set causing some tokens to fail with: "Error retrieving session from token in Authorization header: unable to verify jwt token". The key ID from the failing token matches exactly with the freshly added key.

Expected Behavior

OAuth2 proxy key set should be in sync with the EKS OIDC provider, so that all service account token can be validated.

Current Behavior

The EKS OIDC provider returns "cache-control: max-age=604800" in the http header, go-oidc sets 7 days expiration for each remoteKeySet and does NOT fetch keys even if it cannot find a key in the remoteKeySet meaning that authenticated using a token generated from freshly rotated keys fails with: "Error retrieving session from token in Authorization header: unable to verify jwt token".

Possible Solution

version v2.2.1 of go-oidc library contains a bug regarding key rotation strategy which has been solved in v3.0.0, upgrade dependency.

Version used

v6.1.1 (master uses the same go-oidc version), we've now upgraded to v7.1.3 in preparation for a patch release.

[NickMeves]

I opened a PR with the upgrade, do you mind testing with your environment?

[natefive]

Cheers Nick :), I deployed it for testing but now I don't seem to be able to verify tokens from multiple issuers. We've got EKS and Okta issuer audience pairs set using --extra-jwt-issuers

{"kind":"standard","file":"jwt_session.go:51","msg":"Error retrieving session from token in Authorization header: [unable to verify bearer token, oidc: id token issued by a different provider, expected \"<REDACTED OKTA URL>" got \"<REDACTED EKS URL>", failed to verify signature: failed to verify id token signature]","timestamp":"2021/07/05 10:37:47"}

Error comes from: https://github.com/coreos/go-oidc/blob/15b94d97d90c9455981dbdd90a021d34254cfdaf/oidc/verify.go#L245

[natefive]

Ignore the above ^ comment, when I rolled back to v7.1.3 the same was happening (issuing a new token fixed this). I'm running your version now and will keep you posted, it can take up to a week for a the EKS key rotation issue to occur :) cheers

[natefive]

Pretty confident the patch worked 👍, although because I don't know precisely when the rotation happened there's a slim chance the EKS key rotation could of happened in a 40 minute window between the last service account JWT use and the pod being evicted to another node, but it seems small :) cheers

[NickMeves]

Thanks for circling back! I'll merge the PR

[natefive]

Any ETA on when the next patch release is scheduled for :) cheers

[natefive]

@NickMeves Any update on when a patch release is due? cheers

[JoelSpeed]

We've got a couple more PRs we want to merge before the next release, we are aiming for mid september currently