Azure AD v1: Unable to verify JWT signature: no matching keys

[colemickens]

Hello. I'm looking for some troubleshooting help.

go-oidc doesn't seem to play nice with Azure Active Directory v1 endpoints.

Working: AADv2 endpoints (issuer=https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47 config=https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/.well-known/openid-configuration) + Kubernetes + OIDC

Only partially working: AADv1 endpoints (issuer=https://sts.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/, config=https://sts.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/.well-known/openid-configuration ) + Kubernetes + OIDC

If I only add an id_token then it uses it as-is and it works.

If I also add the refresh_token then I get the verification error shown.

$ k get nodes
Unable to connect to the server: unable to acquire valid JWT: oidc: unable to verify JWT signature: no matching keys

I'm not sure where to look first. For the AADv1 endpoint, the issuer URL is: https://sts.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/ (trailing slash is important) which puts the OIDC config at: https://sts.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/.well-known/openid-configuration

How can I determine if this is a bug in go-oidc or a bug in how AADv1 endpoints are configured?

[colemickens]

Interesting thing to note... when I add the the refresh_token and invoke kubectl... and look at kubeconfig... I can see that the id-token field is partially filled in.

It has the header/body but not the signature/footer component. It literally ends with a period before where the third section should be: b64chunk.b64chunk.

[ericchiang]

So these are the JWKs URIs where go-oidc expects the keys. Can you give us a id_token that you expect to work (maybe an expired one)? Interesting that they're common keys, though the V2 endpoint returns the same thing.

$ curl -s https://sts.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/.well-known/openid-configuration | jq -r .jwks_uri
https://login.windows.net/common/discovery/keys

Interesting thing to note... when I add the the refresh_token and invoke kubectl... and look at kubeconfig... I can see that the id-token field is partially filled in.

That's really weird. Might be a bug on the kubectl auth provider end.

[colemickens]

Interesting thing is that both the as-is token and the refresh_token scenario work when using the AADv2 endpoints -- hence why I filed here instead of against the auth provider.

I'll work on getting an expired token.

[ericchiang]

@colemickens also those keys are expected to rotate. Might want to grab the keys from the keys endpoint when the token is considered valid.

[colemickens]

Oh well, this is awkward. That's the id_token returned verbatim from the v1 endpoint...

[colemickens]

@ericchiang I'm being told that this is expected, as AADv1 is returning an unsigned id_token and that this is okay because of TLS on the connection.

Any thoughts on this? Maybe go-oidc needs modifications to work with unsigned tokens?

[ericchiang]

Any thoughts on this? Maybe go-oidc needs modifications to work with unsigned tokens?

In the Kubernetes context, absolutely not. Kubernetes needs to verify the id_token from arbitrary source. If it's not signed then anyone can provide whatever id_token they want.

[colemickens]

Agreed. Also looks like the JWT client already exposes a way to do this, it would be a modification in the kubectl code to do this --- I'd need to pass none as a SupportedSigningAlgs.

I guess I'll have to follow up internally on the refresh_token business. Thanks for the help here.

[ashb]

Hi - we're seeing something that we think is related to this issue when trying to use Azure AD OIDC with kubernetes - we get this behaviour if I specify a refresh token or not though.

We tried using the login.microsoftonline.com form of ipd URL but when we do we get this error in the kube API server logs:

E0223 16:19:29.164786       1 oidc.go:178] oidc authenticator: failed to fetch provider discovery data: "issuer" in config (https://sts.windows.net/bd5c6713-7399-4b31-be79-78f2d078e543/) does not match provided issuer URL (https://login.microsoftonline.com//bd5c6713-7399-4b31-be79-78f2d078e543/)

Here is an expired id_token:

eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Il9VZ3FYR190TUxkdVNKMVQ4Y2FIeFU3Y090YyIsImtpZCI6Il9VZ3FYR190TUxkdVNKMVQ4Y2FIeFU3Y090YyJ9.eyJhdWQiOiI5MGE4OTg0Yi01NWIyLTQ2YWYtYmQwNi1kNzhjMmU2MzhmZGIiLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC9iZDVjNjcxMy03Mzk5LTRiMzEtYmU3OS03OGYyZDA3OGU1NDMvIiwiaWF0IjoxNDg3ODYxMTM5LCJuYmYiOjE0ODc4NjExMzksImV4cCI6MTQ4Nzg2NTAzOSwiYW1yIjpbInB3ZCJdLCJmYW1pbHlfbmFtZSI6IkJlcmxpbiIsImdpdmVuX25hbWUiOiJBc2giLCJpcGFkZHIiOiIxOTUuMjcuNjAuOTQiLCJuYW1lIjoiQmVybGluLCBBc2giLCJvaWQiOiI2Njg3NDQ0Ny00ZGUyLTRhZTItOWNlNS1lZTU4NDcwYWQ2Y2UiLCJvbnByZW1fc2lkIjoiUy0xLTUtMjEtOTY0MTE0NDg4LTE5MzA3MjEzMDItMTg4NTYyNTE1Ni0yMzc5NjgiLCJwbGF0ZiI6IjUiLCJzdWIiOiI3elROUC1oVzNObnhySjE0ejFTNzBxVS1kZ3MxOE5SdnJCWXBVZGp0OF9vIiwidGlkIjoiYmQ1YzY3MTMtNzM5OS00YjMxLWJlNzktNzhmMmQwNzhlNTQzIiwidW5pcXVlX25hbWUiOiJBc2guQmVybGluQG1uc2NvcnAubmV0IiwidXBuIjoiQXNoLkJlcmxpbkBtbnNjb3JwLm5ldCIsInZlciI6IjEuMCJ9.s8dwgD3a4bJ46yDLAQwTrVLiuUIR5Lv9K_5nwxRyTKpx8Xn6Cy9NUeJ9W7xR1CeqCP-jwktkrafk69nnqsszF8jN9tOH9T9ol7YVTBCx3vFc3pu0Gzq_kSHStaSrUNR2j3wgDiwFHrScQcPqHTZY_JR70mcmGRdFYAlFDk-fHJiv6CovX5M
UXsR1NC8NMKn-9oO0B0rkaQg5oG2QDjPL_rd6NQo2IKvTpiM3RwpUmqC7pWpIjvIgUc3CHicBxXqAEPgpuRPH7uznxnJQ8bEAVJcY9KYzXN-_jUGb85h7v1Dt_LrwtWaMbOHZYKWNGsMEfVQjYJ7NpQ5Qac_3LZpSxQ

[ericchiang]

"issuer" in config (https://sts.windows.net/bd5c6713-7399-4b31-be79-78f2d078e543/) does not match provided issuer URL (https://login.microsoftonline.com//bd5c6713-7399-4b31-be79-78f2d078e543/)

Your issuer is https://sts.windows.net/bd5c6713-7399-4b31-be79-78f2d078e543/ not what you provided to your API server. You need to give it that URL.

[colemickens]

@ashb For AADv1, you need to use https://sts.windows,net/{tenant}/ (needs the trailing slash!) as the Issuer... but note that refreshed id_tokens are not signed and thus won't work with Kubernetes. For AADv2, you need to use https://login.microsoftonline.com/{tenant}/v2. Make sure you use /v2 at the end of the issuer URL to opt into the AADv2 behavior. That will give you a refreshable, signed id_token.

[ericchiang]

@colemickens maybe we should add some docs to the Kubernetes upstream docs :)

[colemickens]

@ericchiang I'd be very happy to contribute some upstream docs, but I'm trying to close the loop on a couple peculiarities around the AADv1/v2 implementations to understand which we should advise for k8s/oidc. In the meantime, for anyone else who winds up here, this may be applicable: https://github.com/colemickens/azure-ad-k8s-oidc-example

But yes, I will plan to upstream the important Azure-y bits of that document to one of the OIDC/auth pages on kubernetes.github.io.

[ericchiang]

@colemickens I'll try to add them if I have some free cycles too. Feel free to assign me if you get to them first.

[ashb]

Thanks both of you. We got most of the way with the existing docs but ran into this error and were a little bit stumped. 👍