[Discussion] EKS Pod Identity

[danielloader]

I'll start with a controversial statement - Since EKS Pod Identity has been launched at re:invent 2023 the value add for creating IRSA roles, passing those IAM roles through to ArgoCD and down into ServiceAccount annotations seems to have diminished.

Is this a fair read on the situation? It's removed the chicken and egg situation I once faced and why I assumed this repository came to exist in the first place.

Open to thoughts on the matter and hopefully start some discourse as someone who has adopted the patterns in this repository and found them to be the only way to make IRSA truly usable in my organisation - but my eyes are on the future and simplification where possible.

[Plork]

You will need to do the association somewhere. You might not need to pass the variable(s) to the helm chart (if you use the correct service name and namespace) but you still need to create the iam and associate it.

Also the application need to use a recent version of the aws sdk to be able to support it if I read correctly.

For me the goal is to remove the going back and forth from TF to yaml. Our idea (still idea) is to use the terraform operator for this. So we can make a crd for a TF module that does the association and make it part of the chart.

[danielloader]

Yeah I've done an internal comparison between tf-controller, Pulumi operator and Crossplane this week so familiar with the play.

Though in those scenarios you still need to pass something into the cluster to be able to "get your bearings" - like the current VPC ID, or the EKS cluster name, or subnets etc, things that allow you to do relative lookups. I've had a decent enough success with just passing in the VPC Id and the EKS cluster name, as you can derive most other things from those terraform data sources...

That being said, I am seeing the value of passing arbitrary strings into my ArgoCD cluster secret to be able to use them in templating in the ApplicationSets, as this repo demonstrated so nicely.

I just see the IRSA/IAM Role ARN/Service Account chain part of this process diminishing with time. Doubly so if they add wildcard support for namespaces and service accounts in the Pod Identity mappings. This entire paradigm has been born out of the limitations we faced and how to surpass them so I'll be happy for the day when they don't exist any more and it all just works.

[csantanapr]

Hi @danielloader thank you for kindly comments on the project.

I have same hopes as you to no needing to pass the IAM role for IRSA, and just do the Mapping between IAM role and Service Account in pure IaC like Terraform/Crossplane/Pulumi/CDK.

The git repos remain useful with examples to answer the question how do I use best practices to use application sets, and deploy managed kubernetes with argocd. I could have called the org "IaC and ArgoCD Best practices" and nothing about "bridge" oh well 😄

The only bridging left is for minor metadata listed in this list and its a good thing that the metadata passed went down:

• Cloudwatch clusterName https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template/blob/main/bootstrap/control-plane/addons/aws/addons-aws-cloudwatch-metrics-appset.yaml#L59
• Fluentbit logs https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template/blob/main/bootstrap/control-plane/addons/aws/addons-aws-fluentbit-appset.yaml#L61-L64
• Fluentbit fargate https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template/blob/main/bootstrap/control-plane/addons/aws/addons-aws-fluentbit-fargate-appset.yaml#L59-L61
• https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template/blob/main/bootstrap/control-plane/addons/aws/addons-aws-gateway-api-controller-appset.yaml#L63-L65
• https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template/blob/main/bootstrap/control-plane/addons/aws/addons-aws-load-balancer-controller-appset.yaml#L59-L60
• https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template/blob/main/bootstrap/control-plane/addons/aws/addons-aws-node-termination-handler-appset.yaml#L59-L61
• https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template/blob/main/bootstrap/control-plane/addons/aws/addons-aws-oss-cluster-autoscaler-appset.yaml#L113-L117
• https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template/blob/main/bootstrap/control-plane/addons/aws/addons-aws-oss-external-dns-appset.yaml#L65-L66
• https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template/blob/main/bootstrap/control-plane/addons/aws/addons-aws-oss-karpenter-appset.yaml#L63-L64
• https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template/blob/main/bootstrap/control-plane/addons/aws/addons-aws-oss-velero-appset.yaml#L65-L73

[csantanapr]

Even if no metadata data is passed I find it useful the examples of showing the ApplicationSet used with cluster generator to show how you can have different clusters for different environments (dev, stage, prod) this could be 100s of clusters using the same repo each with an argocd instance, but all the clusters with same config having different versions of helm charts and corresponding values.yaml

[danielloader]

For sure, this repository has been a fantastic introduction (and I've adopted it at work) for doing environmental configs with ApplicationSets, so thanks for that.

It's a bit cross cutting as I've been doing most of my musing on the Akuity/Kargo discord, if we end up in a position where we can pull helm values from an arbitrary secret and or free form hash map of values on a Stage Kargo Object then we could replicate most of this in Kargo too.

At the very least I'm excited for reducing my IRSA use as 2024 goes on. Guess still waiting on the upstream Terraform modules to adopt it before it's going to be rolled out in my org.

Cross link for anyone else stumbling on this issue: aws-ia/terraform-aws-eks-blueprints-addons#289

[mbuotidem]

Same here, adopted this as well. Thank you for this entire project, its been immensely helpful. I still struggle with having to make two commits, one in the bootstrap and another in the template-repo when I need to turn something else on, but I guess there's not much anyone can do about that.

[csantanapr]

Hi @danielloader the team is looking on implementing EKS Pod identity in the, once we have a v2 I will update gitops-bridge

[csantanapr]

Hi @mbuotidem
Regarding:

I still struggle with having to make two commits, one in the bootstrap and another in the template-repo when I need to turn something else on

Can you provide more details on what is the pain point?
We can brainstorm together on how to alleviate the pain.

[Plork]

On the other hand ... once you automate everything the added value apart from the visibility from EKS console is limited imho.

Cross account permissions (for example external dns with route53 in a different account) become more difficult with pod-identity.

[danielloader]

https://github.com/terraform-aws-modules/terraform-aws-eks-pod-identity It’s here woo

[danielloader]

#55

This has gone a long way to getting there, thanks!