[EKS Prow Cluster] Add karpenter terraform module for eks-prow-build cluster #6895
Conversation
k8s-ci-robot
added
area/infra
|
/assign @xmudrii |
k8s-ci-robot
added
the
size/L
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: koksay The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| @@ -81,6 +82,27 @@ locals { | |||
| } | |||
| ] | |||
|
|
|||
| karpenter_roles = [ | |||
There was a problem hiding this comment.
should we use cluster access entry instead of adding to the aws-auth configmap?
There was a problem hiding this comment.
That's a good question. I tried this code on the Canary cluster first and could not run Karpenter with the access entry. I may have missed something, so I turned it off and continued with the aws-auth file.
| cluster_name = module.eks.cluster_name | ||
| create_access_entry = false | ||
|
|
||
| enable_irsa = true |
There was a problem hiding this comment.
why not use EKS Pod Identity here?
| rolearn = "arn:aws:iam::468814281478:role/AWSReservedSSO_AdministratorAccess_abaef4db15a2c055" | ||
| username = "sso-admins" | ||
| groups = [ | ||
| "eks-cluster-admin" |
There was a problem hiding this comment.
not sure what permissions this group maps to, but similar to above - would cluster access entry work here instead?
| interruptionQueue: Karpenter-prow-canary-cluster | ||
| serviceAccount: | ||
| annotations: | ||
| eks.amazonaws.com/role-arn: "arn:aws:iam::054318140392:role/KarpenterController-20240527081538529900000002" |
There was a problem hiding this comment.
EKS Pod Identity would remove this hardcoded role arn mapping in code
| serviceAccount: | ||
| annotations: | ||
| # terraform state show module.karpenter.aws_iam_role.controller\[0\] | grep " arn " | ||
| eks.amazonaws.com/role-arn: arn:aws:iam::468814281478:role/KarpenterController-20240527081538529900000002 |
There was a problem hiding this comment.
same comment - EKS Pod Identity removes this hardcoding
| @@ -90,6 +101,12 @@ data "aws_iam_policy_document" "eks_apply" { | |||
| "logs:PutRetentionPolicy", | |||
| "logs:TagLogGroup", | |||
| "s3:PutObject", | |||
| "sqs:createqueue", | |||
There was a problem hiding this comment.
Are these permissions required by prow, or by Karpenter? Likewise fir the Eventbridge API permissions
There was a problem hiding this comment.
These are required by the karpenter module, it creates a SQS queue for the event messaging.
Make sure to update the
serviceAccount.annotationsfield in theinfra/aws/terraform/prow-build-cluster/resources/karpenter/flux-hr-karpenter.yamlfile (also ininfra/aws/terraform/prow-build-cluster/resources/karpenter/prod-cluster-values):There will be a follow-up PR to add nodepool and nodeclass configration.