AWS IAM: Support for AWS EKS ServiceAccount roles for CloudWatch and S3 image upload

[patstrom]

What this PR does / why we need it:

This PR replicates (most of) the behavior of aws-sdk-go when fetching a WebIdentityRole such as they are used in IAM roles for service account in EKS.

The reason for the "(most of)" qualifier is that this will only read the environment variables, so if e.g a role_session_name is configured in a configuration file somewhere it won't be picked up. I suspect this won't be much of a problem since all documentation and whatnot for EKS IAM roles for Service Accounts exclusively uses those environment variables.

As discussed in the issue the "proper" way to fix this would probably be to always use the default credentials chain unless a specific way is specified in which case only that way should be used.

I've tried this out in an EKS cluster with a standard setup. In other words I've created a deployment and a service account, linked them together, annotated the service account with eks.amazonaws.com/role-arn and added the necessary trust policies on the role and I can confirm it manages to find the credentials.

Which issue(s) this PR fixes:

Fix #20473

[patstrom]

I realized this change is probably needed in the s3uploader for images as well. Perhaps the AWS configuration should be centralized into a single pkg that both the cloudwatch executor, the image uploader and any potential future interactions can re-use?

[aknuds1]

I made a couple of suggestions for minor improvements. I'll see if I can test this in order to reproduce the issue.

[aknuds1]

Thanks for applying my suggestions, I'm just waiting to get access to an EKS cluster so I can test.

[aknuds1]

Handing this over to @ryantxu since he needs to look it over too.

[patstrom]

@evq you need to sign the cla.

@ryantxu what is the status on this?

[evq]

@patstrom signed

[iptizer]

Is there anything I can do to speed this up a little bit? Hit me!

How I see it @aknuds1 currently has some changes requested. Can I help you testing or something?

[patstrom]

I've already addressed the requested changes. Perhaps @aknuds1 needs to mark them as resolved as well?

[CLAassistant]

All committers have signed the CLA.

[aknuds1]

LGTM, but waiting for @ryantxu's review.

[patstrom]

@ryantxu What is the status on this?

[dnascimento]

@ryantxu When may you review this?

[chancez]

How does this work in combination with configuring the assume role ARN for the cloudwatch datasource? Can we use the EKS service account as the source profile to assume role with the AWS IAM role specified in the datasource? I'm trying to see if once this is fixed if I can use the SA IAM role to allow assuming into a different IAM role per cloudwatch datasource so I can use a single grafana with multiple AWS accounts.

[dnascimento]

@patstromm, could you set webIdentityProvider before remoteCredProvider? It's likely that the EC2 instances have a role attached to it, so this order will prioritise the EC2 instance role over the webIdentity. It's better to try first to use the WebIdentity

[dnascimento]

How does this work in combination with configuring the assume role ARN for the cloudwatch datasource? Can we use the EKS service account as the source profile to assume role with the AWS IAM role specified in the datasource? I'm trying to see if once this is fixed if I can use the SA IAM role to allow assuming into a different IAM role per cloudwatch datasource so I can use a single grafana with multiple AWS accounts.

The EKS service account role can assume a role as well if you define the arn, the code flow is:

grafana/pkg/tsdb/cloudwatch/credentials.go

Line 48 in 42290de

if dsInfo.AuthType == "arn" {

grafana/pkg/tsdb/cloudwatch/credentials.go

Line 64 in 42290de

webIdentityProvider(stsSess),

grafana/pkg/tsdb/cloudwatch/credentials.go

Line 76 in 42290de

resp, err := svc.AssumeRole(params)

@patstrom I tested the code in our systems, could you address the 2 issues above so that this can be reviewed and merged?

[chancez]

@dnascimento yep, I see now, looks like it should work. I'll build this locally and see if I can get it to work with multiple roles. Thanks.

[patstrom]

I currently have a WIP of the "proper" way here, where I add a AWS SDK Default authentication provider that does exactly what you'd expect.

I plan to open a separate MR when it's ready (hopefully later this week).

[ryantxu]

@patstrom -- the new approach looks great. thanks for the effort

[aknuds1]

Synced your branch with Grafana master, to fix CI issues. Hope you don't mind!

[aknuds1]

This took a while, but we finally got it merged - thank you for contributing to Grafana!

[patstrom]

Cheers! 🎉

You're not getting rid of me that easily though. I just opened a new PR with the "proper" way to solve this that also includes some other fixes to the Cloudwatch data source executor.