refactor: IAM Policy improvements on Karpenter Add-on for TFSec compliance rules compliance. #1365

rodrigobersa · 2023-01-25T01:48:47Z

What does this PR do?

Adjustments in data.aws_iam_policy_document.karpenter, on modules/kubernetes-addons/karpenter/data.tf to implement least privilege access policies for IAM Roles for Service Accounts (IRSA).

Motivation

Partially resolves Improve security contexts promoting TFSec integration and usage. #1256
Grant least privilege access for IRSAs
Be compliant with TFSec rules, fixing CRITICAL and HIGH impact findings.

More

Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
Yes, I have added a new example under examples to support my PR
Yes, I have created another PR for add-ons under add-ons repo (if applicable)
Yes, I have updated the docs for this feature
Yes, I ran pre-commit run -a with this PR

Note: Not all the PRs require a new example and/or doc page. In general:

Use an existing example when possible to demonstrate a new addons usage
A new docs page under docs/add-ons/* is required for new a new addon

For Moderators

E2E Test successfully complete before merge?

Additional Notes

Pre Commit & TFSec check

terraform-aws-eks-blueprints git:(2023-01-23-tfsec-kubernetes-addons-karpenter) pre-commit run --file modules/kubernetes-addons/karpenter/*                                                         
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check for merge conflicts................................................Passed
detect private key.......................................................Passed
detect aws credentials...................................................Passed
Terraform fmt............................................................Passed
Terraform docs...........................................................Passed
Terraform validate with tflint...........................................Passed
Terraform validate.......................................................Passed
Terraform validate with tfsec........................(no files to check)Skipped

terraform-aws-eks-blueprints git:(2023-01-23-tfsec-kubernetes-addons-karpenter) tfsec modules/kubernetes-addons/karpenter 
  timings
  ──────────────────────────────────────────
  disk i/o             309.206µs
  parsing              13.882499ms
  adaptation           1.842583ms
  checks               4.759958ms
  total                20.794246ms

  counts
  ──────────────────────────────────────────
  modules downloaded   0
  modules processed    3
  blocks processed     57
  files read           14

  results
  ──────────────────────────────────────────
  passed               27
  ignored              0
  critical             0
  high                 0
  medium               0
  low                  0


No problems detected!

Karpenter validation

Created Karpenter provisioner

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  requirements:
    - key: "node.kubernetes.io/instance-type"
      operator: In
      values: ["m5.large", "t3.large"]
  provider:
    subnetSelector:
      kubernetes.io/cluster/bersr: "shared"
    securityGroupSelector:
      kubernetes.io/cluster/bersr: "owned"
  ttlSecondsAfterEmpty: 30
  limits:
    resources:
      cpu: "20"
  labels:
    karpenter-node: "true"
EOF

Scale deployment to trigger Node scale processess
Scale-out logs

kubectl -n karpenter logs karpenter-645754498f-2svnq

2023-01-20T01:53:51.723Z        DEBUG   Successfully created the logger.
2023-01-20T01:53:51.723Z        DEBUG   Logging level set to: debug
{"level":"info","ts":1674179631.727763,"logger":"fallback","caller":"injection/injection.go:63","msg":"Starting informers..."}
2023-01-20T01:53:51.828Z        DEBUG   controller      waiting for configmaps  {"commit": "f60dacd", "configmaps": ["karpenter-global-settings"]}
2023-01-20T01:53:52.332Z        DEBUG   controller      karpenter-global-settings config "karpenter-global-settings" config was added or updated: settings.Settings{BatchMaxDuration:v1.Duration{Duration:10000000000}, BatchIdleDuration:v1.Duration{Duration:1000000000}}    {"commit": "f60dacd"}
2023-01-20T01:53:52.332Z        DEBUG   controller      karpenter-global-settings config "karpenter-global-settings" config was added or updated: settings.Settings{ClusterName:"bersr", ClusterEndpoint:"https://01AB9AF51D9F0D05EEE7DF7457E5EA49.gr7.us-west-2.eks.amazonaws.com", DefaultInstanceProfile:"bersr-managed-ondemand", EnablePodENI:false, EnableENILimitedPodDensity:true, IsolatedVPC:false, NodeNameConvention:"ip-name", VMMemoryOverheadPercent:0.075, InterruptionQueueName:"", Tags:map[string]string{}}        {"commit": "f60dacd"}
2023-01-20T01:53:52.475Z        DEBUG   controller.aws  discovered region       {"commit": "f60dacd", "region": "us-west-2"}
2023-01-20T01:53:52.481Z        DEBUG   controller.aws  discovered kube dns     {"commit": "f60dacd", "kube-dns-ip": "10.100.0.10"}
2023-01-20T01:53:52.650Z        DEBUG   controller.aws  discovered version      {"commit": "f60dacd", "version": "v0.20.0"}
2023/01/20 01:53:52 Registering 2 clients
2023/01/20 01:53:52 Registering 2 informer factories
2023/01/20 01:53:52 Registering 3 informers
2023/01/20 01:53:52 Registering 6 controllers
2023-01-20T01:53:52.651Z        INFO    controller      Starting server {"commit": "f60dacd", "path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2023-01-20T01:53:52.651Z        INFO    controller      Starting server {"commit": "f60dacd", "kind": "health probe", "addr": "[::]:8081"}
I0120 01:53:52.752642       1 leaderelection.go:248] attempting to acquire leader lease karpenter/karpenter-leader-election...
2023-01-20T01:53:52.815Z        INFO    controller      Starting informers...   {"commit": "f60dacd"}
2023-01-20T01:53:53.045Z        INFO    controller.aws.pricing  updated spot pricing with instance types and offerings  {"commit": "f60dacd", "instance-type-count": 607, "offering-count": 2095}
2023-01-20T01:53:55.057Z        INFO    controller.aws.pricing  updated on-demand pricing       {"commit": "f60dacd", "instance-type-count": 606}
I0120 01:54:09.549191       1 leaderelection.go:258] successfully acquired lease karpenter/karpenter-leader-election
2023-01-20T01:54:09.549Z        INFO    controller.provisioner  starting controller     {"commit": "f60dacd"}
2023-01-20T01:54:09.550Z        INFO    controller.deprovisioning       starting controller     {"commit": "f60dacd"}
2023-01-20T01:54:09.550Z        INFO    controller.metricscraper        starting controller     {"commit": "f60dacd"}
2023-01-20T01:54:09.550Z        INFO    controller      Starting EventSource    {"commit": "f60dacd", "controller": "node-state", "controllerGroup": "", "controllerKind": "Node", "source": "kind source: *v1.Node"}
2023-01-20T01:54:09.551Z        INFO    controller      Starting Controller     {"commit": "f60dacd", "controller": "node-state", "controllerGroup": "", "controllerKind": "Node"}
2023-01-20T01:54:09.551Z        INFO    controller      Starting EventSource    {"commit": "f60dacd", "controller": "provisioning", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2023-01-20T01:54:09.551Z        INFO    controller      Starting Controller     {"commit": "f60dacd", "controller": "provisioning", "controllerGroup": "", "controllerKind": "Pod"}
2023-01-20T01:54:09.552Z        INFO    controller      Starting EventSource    {"commit": "f60dacd", "controller": "provisionerstate", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "source": "kind source: *v1alpha5.Provisioner"}
2023-01-20T01:54:09.552Z        INFO    controller      Starting Controller     {"commit": "f60dacd", "controller": "provisionerstate", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner"}
2023-01-20T01:54:09.553Z        INFO    controller      Starting EventSource    {"commit": "f60dacd", "controller": "pod-state", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2023-01-20T01:54:09.553Z        INFO    controller      Starting Controller     {"commit": "f60dacd", "controller": "pod-state", "controllerGroup": "", "controllerKind": "Pod"}
2023-01-20T01:54:09.553Z        INFO    controller      Starting EventSource    {"commit": "f60dacd", "controller": "termination", "controllerGroup": "", "controllerKind": "Node", "source": "kind source: *v1.Node"}
2023-01-20T01:54:09.553Z        INFO    controller      Starting Controller     {"commit": "f60dacd", "controller": "termination", "controllerGroup": "", "controllerKind": "Node"}
2023-01-20T01:54:09.554Z        INFO    controller      Starting EventSource    {"commit": "f60dacd", "controller": "node", "controllerGroup": "", "controllerKind": "Node", "source": "kind source: *v1.Node"}
2023-01-20T01:54:09.554Z        INFO    controller      Starting EventSource    {"commit": "f60dacd", "controller": "node", "controllerGroup": "", "controllerKind": "Node", "source": "kind source: *v1alpha5.Provisioner"}
2023-01-20T01:54:09.554Z        INFO    controller      Starting EventSource    {"commit": "f60dacd", "controller": "node", "controllerGroup": "", "controllerKind": "Node", "source": "kind source: *v1.Pod"}
2023-01-20T01:54:09.554Z        INFO    controller      Starting EventSource    {"commit": "f60dacd", "controller": "node", "controllerGroup": "", "controllerKind": "Node", "source": "channel source: 0xc0011bf7c0"}
2023-01-20T01:54:09.554Z        INFO    controller      Starting Controller     {"commit": "f60dacd", "controller": "node", "controllerGroup": "", "controllerKind": "Node"}
2023-01-20T01:54:09.554Z        INFO    controller      Starting EventSource    {"commit": "f60dacd", "controller": "provisionermetrics", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "source": "kind source: *v1alpha5.Provisioner"}
2023-01-20T01:54:09.554Z        INFO    controller      Starting Controller     {"commit": "f60dacd", "controller": "provisionermetrics", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner"}
2023-01-20T01:54:09.555Z        INFO    controller      Starting EventSource    {"commit": "f60dacd", "controller": "podmetrics", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2023-01-20T01:54:09.555Z        INFO    controller      Starting Controller     {"commit": "f60dacd", "controller": "podmetrics", "controllerGroup": "", "controllerKind": "Pod"}
2023-01-20T01:54:09.555Z        INFO    controller      Starting EventSource    {"commit": "f60dacd", "controller": "inflightchecks", "controllerGroup": "", "controllerKind": "Node", "source": "kind source: *v1.Node"}
2023-01-20T01:54:09.555Z        INFO    controller      Starting Controller     {"commit": "f60dacd", "controller": "inflightchecks", "controllerGroup": "", "controllerKind": "Node"}
2023-01-20T01:54:09.556Z        INFO    controller      Starting EventSource    {"commit": "f60dacd", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "source": "kind source: *v1alpha5.Provisioner"}
2023-01-20T01:54:09.556Z        INFO    controller      Starting EventSource    {"commit": "f60dacd", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "source": "kind source: *v1.Node"}
2023-01-20T01:54:09.556Z        INFO    controller      Starting Controller     {"commit": "f60dacd", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner"}
2023-01-20T01:54:09.557Z        INFO    controller.interruption starting controller     {"commit": "f60dacd"}
2023-01-20T01:54:09.557Z        DEBUG   controller.aws  hydrating the launch template cache     {"commit": "f60dacd", "tag-key": "karpenter.k8s.aws/cluster", "tag-value": "bersr"}
2023-01-20T01:54:09.606Z        DEBUG   controller.aws  finished hydrating the launch template cache    {"commit": "f60dacd", "tag-key": "karpenter.k8s.aws/cluster", "tag-value": "bersr", "item-count": 0}
2023-01-20T01:54:09.651Z        INFO    controller      Starting workers        {"commit": "f60dacd", "controller": "node-state", "controllerGroup": "", "controllerKind": "Node", "worker count": 10}
2023-01-20T01:54:09.652Z        INFO    controller      Starting workers        {"commit": "f60dacd", "controller": "provisioning", "controllerGroup": "", "controllerKind": "Pod", "worker count": 10}
2023-01-20T01:54:09.847Z        INFO    controller      Starting workers        {"commit": "f60dacd", "controller": "pod-state", "controllerGroup": "", "controllerKind": "Pod", "worker count": 10}
2023-01-20T01:54:09.847Z        INFO    controller      Starting workers        {"commit": "f60dacd", "controller": "podmetrics", "controllerGroup": "", "controllerKind": "Pod", "worker count": 1}
2023-01-20T01:54:09.973Z        INFO    controller      Starting workers        {"commit": "f60dacd", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "worker count": 10}
2023-01-20T01:54:10.005Z        INFO    controller      Starting workers        {"commit": "f60dacd", "controller": "provisionerstate", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "worker count": 10}
2023-01-20T01:54:10.008Z        INFO    controller      Starting workers        {"commit": "f60dacd", "controller": "termination", "controllerGroup": "", "controllerKind": "Node", "worker count": 10}
2023-01-20T01:54:10.008Z        INFO    controller      Starting workers        {"commit": "f60dacd", "controller": "provisionermetrics", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "worker count": 1}
2023-01-20T01:54:10.008Z        INFO    controller      Starting workers        {"commit": "f60dacd", "controller": "node", "controllerGroup": "", "controllerKind": "Node", "worker count": 10}
2023-01-20T01:54:10.014Z        INFO    controller      Starting workers        {"commit": "f60dacd", "controller": "inflightchecks", "controllerGroup": "", "controllerKind": "Node", "worker count": 10}
2023-01-20T01:54:11.097Z        DEBUG   controller.deprovisioning       discovered EC2 instance types   {"commit": "f60dacd", "instance-type-count": 598}
2023-01-20T01:54:11.164Z        DEBUG   controller.deprovisioning       discovered subnets      {"commit": "f60dacd", "subnets": ["subnet-03c3c7db3ee2ca138 (us-west-2a)", "subnet-0e9737a3c91475621 (us-west-2c)", "subnet-01bde363942e626fd (us-west-2b)", "subnet-02243d0d7eddd4883 (us-west-2d)"]}
2023-01-20T01:54:11.433Z        DEBUG   controller.deprovisioning       discovered EC2 instance types zonal offerings for subnets       {"commit": "f60dacd", "subnet-selector": "{\"kubernetes.io/cluster\":\"bersr\"}"}
2023-01-20T01:54:17.083Z        DEBUG   controller.provisioner  351 out of 598 instance types were excluded because they would breach provisioner limits        {"commit": "f60dacd"}
2023-01-20T01:54:17.090Z        INFO    controller.provisioner  found provisionable pod(s)      {"commit": "f60dacd", "pods": 22}
2023-01-20T01:54:17.090Z        INFO    controller.provisioner  computed new node(s) to fit pod(s)      {"commit": "f60dacd", "nodes": 1, "pods": 22}
2023-01-20T01:54:17.090Z        INFO    controller.provisioner  launching node with 22 pods requesting {"cpu":"855m","memory":"420Mi","pods":"29"} from types m5.large, m5.2xlarge      {"commit": "f60dacd", "provisioner": "default"}
2023-01-20T01:54:17.195Z        DEBUG   controller.provisioner.cloudprovider    discovered security groups      {"commit": "f60dacd", "provisioner": "default", "security-groups": ["sg-07fd1ef2443b3ba25"]}
2023-01-20T01:54:17.205Z        DEBUG   controller.provisioner.cloudprovider    discovered kubernetes version   {"commit": "f60dacd", "provisioner": "default", "kubernete-version": "1.24"}
2023-01-20T01:54:17.235Z        DEBUG   controller.provisioner.cloudprovider    discovered new ami      {"commit": "f60dacd", "provisioner": "default", "ami": "ami-06bb00841fcd76aa4", "query": "/aws/service/eks/optimized-ami/1.24/amazon-linux-2/recommended/image_id"}
2023-01-20T01:54:17.389Z        DEBUG   controller.provisioner.cloudprovider    created launch template {"commit": "f60dacd", "provisioner": "default", "launch-template-name": "Karpenter-bersr-6436745216526544972", "launch-template-id": "lt-0bb25bd13b9bc4f49"}
2023-01-20T01:54:19.163Z        INFO    controller.provisioner.cloudprovider    launched new instance   {"commit": "f60dacd", "provisioner": "default", "launched-instance": "i-0135bd5c83f0d3c6a", "hostname": "ip-172-31-24-172.us-west-2.compute.internal", "type": "m5.large", "zone": "us-west-2b", "capacity-type": "on-demand"}

kubectl describe node ip-172-31-24-172.us-west-2.compute.internal
Name:               ip-172-31-24-172.us-west-2.compute.internal
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=m5.large
                    beta.kubernetes.io/os=linux
                    billing-team=my-team
                    failure-domain.beta.kubernetes.io/region=us-west-2
                    failure-domain.beta.kubernetes.io/zone=us-west-2b
                    k8s.io/cloud-provider-aws=0753ed54af835624ce66ff8bae610d4e
                    karpenter.k8s.aws/instance-ami-id=ami-06bb00841fcd76aa4
                    karpenter.k8s.aws/instance-category=m
                    karpenter.k8s.aws/instance-cpu=2
                    karpenter.k8s.aws/instance-family=m5
                    karpenter.k8s.aws/instance-generation=5
                    karpenter.k8s.aws/instance-hypervisor=nitro
                    karpenter.k8s.aws/instance-memory=8192
                    karpenter.k8s.aws/instance-pods=29
                    karpenter.k8s.aws/instance-size=large
                    karpenter.sh/capacity-type=on-demand
                    karpenter.sh/initialized=true
                    karpenter.sh/provisioner-name=default
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-172-31-24-172.us-west-2.compute.internal
                    kubernetes.io/os=linux
                    node.kubernetes.io/instance-type=m5.large
                    topology.ebs.csi.aws.com/zone=us-west-2b
                    topology.kubernetes.io/region=us-west-2
                    topology.kubernetes.io/zone=us-west-2b

Scale-in logs

kubectl -n karpenter logs karpenter-645754498f-2svnq

2023-01-20T02:01:18.886Z        INFO    controller.node added TTL to empty node {"commit": "f60dacd", "node": "ip-172-31-24-172.us-west-2.compute.internal"}
2023-01-20T02:01:49.054Z        INFO    controller.deprovisioning       deprovisioning via emptiness delete, terminating 1 nodes ip-172-31-24-172.us-west-2.compute.internal/m5.large/on-demand {"commit": "f60dacd"}
2023-01-20T02:01:49.113Z        INFO    controller.termination  cordoned node   {"commit": "f60dacd", "node": "ip-172-31-24-172.us-west-2.compute.internal"}
2023-01-20T02:01:49.305Z        INFO    controller.termination  deleted node    {"commit": "f60dacd", "node": "ip-172-31-24-172.us-west-2.compute.internal"}
2023-01-20T02:03:52.827Z        DEBUG   controller.aws  deleted launch template {"commit": "f60dacd"}

…ons/karpenter.

…rpenter

bryantbiggs · 2023-01-28T12:18:07Z

modules/kubernetes-addons/karpenter/data.tf

@@ -1,20 +1,16 @@
 data "aws_arn" "queue" {
-  count = var.enable_spot_termination_handling ? 1 : 0
+  count = var.sqs_queue_arn != "" ? 1 : 0


we should leave these as is - comparison checking is problematic. For example, if someone sets this value as null, things will fail

Yes, you're right. In fact I don't really remember changing this count, but I anyway I reverted it.

…iance rules compliance. (aws-ia#1365) Co-authored-by: Rodrigo Bersa <bersr@amazon.com> Co-authored-by: Bryant Biggs <bryantbiggs@gmail.com> resolves undefined

bersr-aws and others added 3 commits January 23, 2023 15:31

IAM Policy adjustments for TFSec compliance on modules/kubernetes-add…

0c0d2b9

…ons/karpenter.

Merge branch 'aws-ia:main' into 2023-01-23-tfsec-kubernetes-addons-ka…

23c6de0

…rpenter

pre-commit: fix fmt, add rule for unused variable

32218db

rodrigobersa had a problem deploying to EKS Blueprints Test January 25, 2023 01:48 — with GitHub Actions Error

rodrigobersa marked this pull request as ready for review January 28, 2023 02:58

rodrigobersa requested a review from a team as a code owner January 28, 2023 02:58

bryantbiggs reviewed Jan 28, 2023

View reviewed changes

Adjusting count for aws_arn.queue data source.

50dc4c4

rodrigobersa had a problem deploying to EKS Blueprints Test January 30, 2023 00:35 — with GitHub Actions Error

Merge branch 'main' into 2023-01-23-tfsec-kubernetes-addons-karpenter

71a933b

bryantbiggs temporarily deployed to EKS Blueprints Test January 30, 2023 23:35 — with GitHub Actions Inactive

bryantbiggs approved these changes Feb 11, 2023

View reviewed changes

bryantbiggs merged commit 32d6a9f into aws-ia:main Feb 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: IAM Policy improvements on Karpenter Add-on for TFSec compliance rules compliance. #1365

refactor: IAM Policy improvements on Karpenter Add-on for TFSec compliance rules compliance. #1365

rodrigobersa commented Jan 25, 2023 •

edited

Loading

bryantbiggs Jan 28, 2023

rodrigobersa Jan 30, 2023

refactor: IAM Policy improvements on Karpenter Add-on for TFSec compliance rules compliance. #1365

refactor: IAM Policy improvements on Karpenter Add-on for TFSec compliance rules compliance. #1365

Conversation

rodrigobersa commented Jan 25, 2023 • edited Loading

What does this PR do?

Motivation

More

For Moderators

Additional Notes

Pre Commit & TFSec check

Karpenter validation

bryantbiggs Jan 28, 2023

Choose a reason for hiding this comment

rodrigobersa Jan 30, 2023

Choose a reason for hiding this comment

rodrigobersa commented Jan 25, 2023 •

edited

Loading