Proposal: ODIC/Oauth2 client for Remote Write.

[bwplotka]

Currently, it's extremely common to have an OIDC/Oauth2 based server auth, especially for SaaS offerings.

This is why I would propose to add OIDC/Oauth2 client to Prometheus remote write endpoint. Those protocols are well known and well used, so I would expect high usage of those.

I searched through docs and couldn't find the related discussion so starting one here.

I know you can leverage Prometheus by replacing the bearer token in the configuration of remote write (https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) but this request is to add native support.

By any chance, is there any sidecar that already integrates with Prometheus through config or as a front proxy?

[bwplotka]

Mhm, I can answer myself, it's a no for current time being: https://www.robustperception.io/prometheus-security-authentication-authorization-and-encryption so moving to mailing list to look for potential sidecars 🤗

[bwplotka]

https://groups.google.com/g/prometheus-users/c/LoVWVwNhu6o

[brian-brazil]

From previously looking into these for other stuff, my understanding of these providers is that you need to implement code for each one specifically (i.e. one for Google, one for Facebook etc.) and these flows also require human interaction for initial setup. I don't think that Oauth2 is compatible with Prometheus's architecture or deployment model, you'd need to have something else provide the token to Prometheus.

[roidelapluie]

Yes that is possible, but you have that already. To have this working, you need a oauth token, as Prometheus is a server, you would generate a service token in Keycloak, which does not change and can use the existing Token or Token file options we have.

[roidelapluie]

https://wjw465150.gitbooks.io/keycloak-documentation/content/server_admin/topics/clients/oidc/service-accounts.html

It turns out that we would still need to fetch the token.

[bwplotka]

The idea would be to just use OIDC client to do so like https://github.com/bwplotka/oidc

It works against any OIDC compatible backends

[csmarchbanks]

OIDC is common enough that I think it is worthwhile adding native support for service accounts. It is certainly possible as I have implemented OIDC in a remote-write adapter and fluentd plugins before.

[roidelapluie]

#287 #293 will solve this.

[LeviHarrison]

#287, #293, and #294 have been merged, as well as prometheus/prometheus#8761, so I think this is completed.

[roidelapluie]

It is, thanks!

[nicoche]

Hi!

First of all, thanks for your work on Prometheus.

I have a similar use case as prometheus/prometheus#9240: I want to remote_write to a target protected by Identity Aware Proxy (IAP). I think it's OIDC (https://cloud.google.com/iap/docs/authentication-howto#obtaining_an_oidc_token_for_the_default_service_account).
In prometheus/prometheus#9240, it is mentioned that the current implementation should cover that use case. I'm not sure that is true.

The actual call to retrieve an id token is curl -X POST https://oauth2.googleapis.com/token -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=XXX". A few things look missing with the current implementation for that call to happen:

• The documentation mentions that it does OAuth 2.0 authentication using the client_credentials grant type. I believe that we would need to set it to urn:ietf:params:oauth:grant-type:jwt-bearer
• It does not look like audience can be set
• Iss neither

My understanding is that it's thus not possible to remote_write using OIDC.

• Does that look correct to you? I'm no OAuth2 expert, so I could be wrong.
• If it is not correct, how would that work? 🤔
• If it is, would you be open to add OIDC to prometheus remote_write configuration? I'd be happy to help

For now, I'll probably run a proxy alongside my remote_writer but I would be happy if we could have a native solution in Prometheus.

Let me know what you think!

[LeviHarrison]

My understanding is that it's thus not possible to remote_write using OIDC.

I did a little bit of digging to try to figure out if that is a definitive statement, and I found this, which looks like it would work with our implementation (although I haven't tested it), so maybe OIDC is possible. But, in the case of GCP IAP, it is not, yes.

would you be open to add OIDC to prometheus remote_write configuration?

I definitely think it would be reasonable to add a more proper and standard OIDC client if there is a use for it. What I'm wondering though is if it would actually solve your particular use case with GCP. In the example in the GCP documentation, a GCP-specific library is used, so I'm not sure if it would would with generic client like the one that is being proposed.

[nicoche]

Hi @LeviHarrison!

Sorry for the delay and thanks for taking the time to look into that.

I took some time to investigate. Here are my findings:

• You are right regarding the fact that GCP IAP cannot work atm
• GCP IAP does not actually use OIDC but JSON Web Token (JWT) Profile for OAuth 2.0 Client Authentication and Authorization Grants as defined in RFC 7523
• According to this article, RFC7523 is profiled (i.e., further specified) by OIDC 1.0

So that would mean that compliance to GCP IAP may not be achieved by a generic oidc client. However, it can be achieved by an oauth2 client, given enough flexibility in its configuration: we would need the ability to set grant_type and additional claims.

From here, if you are willing to support that flow, the solutions I could think of would be:

1. Provide a GCP specific configuration block. Similar to what has been done for aws sigv4 if I understood correctly
2. Provide a generic RFC7523 configuration block
3. Extend the oauth2 configuration block so it could fit advanced use cases like RFC7523

Solution 1. would be the more straightforward, but I understand the concerns about adding a non-std dependency. The given library seems to however just be a wrapper around https://pkg.go.dev/golang.org/x/oauth2 to hydrate a standard jwt.Config (https://pkg.go.dev/golang.org/x/oauth2@v0.0.0-20220608161450-d0670ef3b1eb/jwt#Config) from a GCP service account with the exception of a custom claim added; so the overhead is relatively low. We could even pick only the few relevant chunks of code if we don't want to import it
Solutions 2 and 3 would work well, but require appropriate documentation for people to understand that the conf blocks refer to GCP (or other). Also, users would need to parse themselves their GCP service accounts. On the bright side, they are both vendor agnostic solution.

Let me know what you think! I'd love to see this move forward

[LeviHarrison]

Hi there. First I'd just like to say thanks for putting that thorough comment together, and I really apologize for the delayed response. We were discussing this at one point but never came to a conclusion.

My main issue with adding native or quasi-native (in the case of options 2 and 3) support for IAP in remote-write is adoption. For option 1, you're obsoletely right that AWS sigv4 has been added in the same manner you're proposing GCP IAP to be. One of the main reasons for having sigv4 though is that it's integral to how metrics are sent to Amazon Managed Service for Prometheus: from an official Prometheus binary using sigv4 to authenticate. As far as I know, IAP isn't necessitated by any (I assume) highly-adopted remote-write backend like AMP.

As for option 2, I feel that would be the best solution if we were to go ahead with native support. It allows for more authentication providers than just IAP, which in turn could make it useful to more users. But still, based on my quick research, RFC7523 is not that widely used.

And for the option 3, I think that a Prometheus's oauth2 config benefits from the simplicity of only supporting client credentials, and I wouldn't want to change that.

To sum it up, I'm worried that native IAP support would not be used by enough people to make it worth implementing and supporting in the future when just using a proxy is a viable (although less convenient) option. This isn't necessarily a final decision though, just my opinion.

cc @roidelapluie