Latest karpenter upgrade causes crash due to list/watch configmap forbidden

[benjefferies]

Description

There are versioning issues with the latest versions of karpenter. This causes issues with the service account. See this issue for more details

E0308 10:50:37.563419       1 reflector.go:140] k8s.io/client-go@v0.26.6/tools/cache/reflector.go:169: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:karpenter:karpenter" cannot list resource "configmaps" in API group "" in the namespace "karpenter"
W0308 10:51:15.861739       1 reflector.go:424] k8s.io/client-go@v0.26.6/tools/cache/reflector.go:169: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:karpenter:karpenter" cannot list resource "configmaps" in API group "" in the namespace "karpenter"
E0308 10:51:15.861856       1 reflector.go:140] k8s.io/client-go@v0.26.6/tools/cache/reflector.go:169: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:karpenter:karpenter" cannot list resource "configmaps" in API group "" in the namespace "karpenter"
W0308 10:51:16.886333       1 reflector.go:424] k8s.io/client-go@v0.26.6/tools/cache/reflector.go:169: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:karpenter:karpenter" cannot list resource "configmaps" in API group "" in the namespace "karpenter"
E0308 10:51:16.886371       1 reflector.go:140] k8s.io/client-go@v0.26.6/tools/cache/reflector.go:169: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:karpenter:karpenter" cannot list resource "configmaps" in API group "" in the namespace "karpenter"
W0308 10:51:18.990191       1 reflector.go:424] k8s.io/client-go@v0.26.6/tools/cache/reflector.go:169: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:karpenter:karpenter" cannot list resource "configmaps" in API group "" in the namespace "karpenter"
E0308 10:51:18.990235       1 reflector.go:140] k8s.io/client-go@v0.26.6/tools/cache/reflector.go:169: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:karpenter:karpenter" cannot list resource "configmaps" in API group "" in the namespace "karpenter"

• ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

Before you submit an issue, please perform the following first:

1. Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
2. Re-initialize the project root to pull down modules: terraform init
3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

• Module version [Required]: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons.git?ref=345ce9f

• Terraform version: 1.6.0

• Provider version(s):

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.47"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.20"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.9"
    }
    bcrypt = {
      source  = "viktorradnai/bcrypt"
      version = ">= 0.1.2"
    }
    random = {
      source  = "hashicorp/random"
      version = ">= 3.1"
    }
  }

Reproduction Code [Required]

Steps to reproduce the behavior:

Deploy karpenter pointing to latest helm chart

Expected behaviour

Karpenter runs ok

Actual behaviour

Karpenter crashes with service account permission issues

Terminal Output Screenshot(s)

Additional context

[bryantbiggs]

ref #366 (comment)