Allow auth using IAM Roles for Service Accounts on EKS

[hasheddan]

Description of your changes

This is initial implementation of allowing an AWS Provider to authenticate using IAM roles for Service Accounts when running crossplane and stack-aws on EKS. A few things to note on this early WIP:

• Auth is only enabled for RDSInstance here, but other resources will look essentially the same
• The AWS SDK has been update from v0.5.0 to v0.19.0, which requires context to be explicitly passed to .Send() methods. I have updated across all controllers and clients, but am just passing context.TODO() into those controllers that are not yet using the managed reconciler. I think this should be addressed in a separate PR.
• Because aws-sdk-go-v2 (which is what we use in stack-aws) does not yet have the functionality implemented in v1 here, I am doing STS auth manually. However, there is indication that v2 will have the new features soon. I normally would say we should wait until it is contributed upstream before adding this feature, but given that we have an explicit request here and we can easily update to consume v2 implementation without changing our API, I think it would be reasonable to go ahead and move forward.
• This has been tested successfully on EKS, but does require some manual configuration. We can likely start with just documenting how to set it up (since early users will likely already be familiar), then actually provide it as a deployment option in the future.
• The current implementation will need some clean-up but I would like to open conversation before fixing any stylistic changes.

Fixes #109 & #110

/cc @phanimullapudi @mfacenet

Checklist

I have:

• Run make reviewable to ensure this PR is ready for review.
• Ensured this PR contains a neat, self documenting set of commits.
• Updated any relevant documentation, examples, or release notes.
• Updated the dependencies in app.yaml to include any new role permissions.

[upbound-bot]

[muvaf]

I think the fact that authentication is done per pod doesn't really fit how Crossplane works in general. It essentially means you can only use one SA-enabled provider for the whole Crossplane cluster because there is only 1 stack-aws in the cluster that watches all AWS managed resources. @hasheddan Is my understanding correct?

Is there a way to overcome this? I image that if we had a controller for Provider type whose purpose is:

• whenever an SA-enabled Provider is created with a field maybe spec.iamServiceAccount, create a new Pod with spec.iamServiceAccount service account attached that will fetch the actual credentials data via AssumeRoleWithWebIdentityRequest.
• Then we could:
  • either possibly store that into a secret that is referred by the Provider, hence stack-aws won't be even changed. But that might be against why user needs to do it in the first place; they don't want to manage the credentials.
  • or in each reconcile stack-aws can send a HTTP query to that Pod for credentials and use the retrieved data for requests in that reconcile. So, the Pod that was created for SA-enabled provider would essentially act like a credential provider.

Another possibility is that when we see an SA-enabled provider, we could somehow create another stack-aws Deployment which has the necessary service account and it'd reconcile only the resources with spec.providerRef == SA-enabled provider that this stack-aws pod is assigned to . But this'd complicate the stack stuff.

[hasheddan]

@muvaf thanks for your thoughts here! Yes, you do understand correctly. While this is a different pattern, it is less about trying to enable using service accounts in general and more about enabling using service accounts and IAM roles the way that EKS does internally. I agree that most Crossplane users will want to use multiple providers, which you could still do here by managing a "core" provider that uses this service account model, and then letting other providers that reference secrets exist as well (potentially with finer scoped permissions).

In general, I think your points are valuable for a conversation about how auth works across Crossplane. It might be worth exploring if the current secret model is sustainable long-term. However, I do not think enabling the functionality in this PR hurts our authentication story, and it does enable users who prefer this pattern to utilize it.

[negz]

@hasheddan This direction looks good to me. I know you said you wanted a review before we get into style details, but I figured I'd leave a few now rather than later. :)

[negz]

What's the reason behind this interface change?

[hasheddan]

The interfaces were renamed somewhere in the bump from v0.5 to v0.19 in the AWS SDK

[negz]

Ditto here - what's the context for this change?

[hasheddan]

Same as above

[negz]

Style nit: I'd prefer an alternate NewServiceAccountClient or a varaidic option like UsePodServiceAccount over this boolean argument.

[negz]

I think the fact that authentication is done per pod doesn't really fit how Crossplane works in general.

@muvaf I agree that this is pretty different from how Crossplane is typically configured, but I do think it's a reasonable MVP to solve the immediate problems a few folks have come to us with. It sounds like some orgs aren't permitted to expose their credentials as a Kubernetes Secret for security reasons; this matches what the folks at AWS have told us is considered best practice. I like that at least our Provider resources have to opt into the functionality, so there's a little gating around whether services get to use the stack pod's credentials.

[muvaf]

Alright, but let's make sure this meets their requirements. @hasheddan I assume they use credentials only in this way. Are they OK with using only one IAM role for resource provisioning in a given cluster? They may think that they can deploy more than one Crossplane instance to the same cluster, which is not possible. So, it'd be great if we can confirm what we give is what they want.

Ideally, I'd like us to discuss the final place we'd like to be in and how we would upgrade the clusters working with this MVP to that final place. Not really in a defined and planned way but at least very general path like when we do X solution, we can migrate the existing setups like that. But given that even if we mess up with auth and require reconfiguration, no resource will be modified/changed, so, it should be alright to not have an idea about that at the moment.

[hasheddan]

@muvaf that’s a great point! @phanimullapudi do the constraints mentioned above still slow for the functionality you desire?

[hasheddan]

Note: PR to implement similar functionality in aws-sdk-go-v2 has been opened but is in draft.

[phanimullapudi]

@muvaf that’s a great point! @phanimullapudi do the constraints mentioned above still slow for the functionality you desire?

Yes using one single IAM role should work for us.

[upbound-bot]

[negz]

LGTM! Thanks for driving this @hasheddan. I left a few comments, but nothing that needs to block merging. Let me know whether you want to address them or go straight to merge.

[hasheddan]

@negz thanks for review! Should be ready for merge now ✔️

[upbound-bot]

[alemairebe]

Could someone please document how to use this feature ?
Thanks !

[hasheddan]

@alemairebe I have added a doc describing how to enable this feature. The v0.9 release of Crossplane will make this even easier. Please let me know if you have any further questions I can help with :)

[alemairebe]

Thank you for this quick answer !
It already helps a lot, my current use-case is to deploy and configure EKS (iam/userdata/asg/...) with TF and use crossplane to abstract access to DB. Could you also document the minimal policy needed by crossplane to handled only RDS ?
Is it planned to enable crossplane to create an IAM role which would be used by the app pod in the stack to access the DB ?

[muvaf]

@alemairebe So, you'd like to use Crossplane to create an IAM Role with a few policies attached and then assume it in the app pod to access to the RDS. Is my understanding correct?

You can create the role via IAMRole CR and attach the policy via IAMRolePolicyAttachment CR (you can find examples in stacks guide or official API reference, too).

How do you plan to enable pod to assume this role? By attaching an annotation on its service account?

[alemairebe]

@muvaf Yes, I would like to use Crossplane to :

1. create an RDS instance with IAMDatabaseAuthentication enabled
2. create a IAM role to access the RDS instance
3. map the IAM role to the masterUsername
4. create a service account with the right annotation to assume the role from a pod

[muvaf]

@alemairebe It should be possible to make this scenario work in Crossplane except for the policy creation part that needs to be done for the IAM role created. I'm working on a set of YAMLs to see how it'd unfold e2e. Will let you know when it's ready.