subnetSelector and securityGroupSelector expect " or n, but found [

[FernandoMiguel]

Version

Karpenter: "0.10.1"

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
WARNING: version difference between client (1.24) and server (1.22) exceeds the supported minor version skew of +/-1
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.6-eks-14c7a48", GitCommit:"35f06c94ad99b78216a3d8e55e04734a85da3f7b", GitTreeState:"clean", BuildDate:"2022-04-01T03:18:05Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

Expected Behavior

to be able to use a list in the selector, like it does in the AZs, to use jsonencode instead of join in the code below

Actual Behavior

│ Error: default failed to run apply: error when applying patch:
│ {
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"karpenter.sh/v1alpha5\",\"kind\":\"Provisioner\",\"metadata\":{\"annotations\":{},\"name\":\"default\"},\"spec\":{\"limits\":{\"resources\":{\"cpu\":1000}},\"provider\":{\"instanceProfile\":\"KarpenterNodeInstanceProfile--precise-mako\",\"securityGroupSelector\":{\"aws-ids\":\"sg-XXXX, sg-XXXX, sg-XXXX\"},\"subnetSelector\":{\"aws-ids\":[\"subnet-XXXXX\",\"subnet-XXXXX\",\"subnet-XXXXX\",\"subnet-XXXXX\",\"subnet-XXXXX\"]},\"tags\":{\"BusinessUnit\":\"cts\",\"Component\":\"eks-modules\",\"Environment\":\"dev\",\"OWNER\":\"Fernando\",\"Product\":\"XXX\",\"SourceCode\":\"https://gitXXX\",\"Team\":\"XXXX\",\"TerraformCreated\":\"true\"}},\"requirements\":[{\"key\":\"topology.kubernetes.io/zone\",\"operator\":\"In\",\"values\":[\"us-east-1a\",\"us-east-1b\",\"us-east-1c\",\"us-east-1d\",\"us-east-1f\"]},{\"key\":\"karpenter.sh/capacity-type\",\"operator\":\"In\",\"values\":[\"spot\",\"on-demand\"]},{\"key\":\"node.kubernetes.io/instance-type\",\"operator\":\"In\",\"values\":[\"c6a.large\",\"c6i.large\"]},{\"key\":\"kubernetes.io/arch\",\"operator\":\"In\",\"values\":[\"arm64\",\"amd64\"]}],\"topologySpreadConstraints\":[{\"maxSkew\":1,\"topologyKey\":\"topology.kubernetes.io/zone\",\"whenUnsatisfiable\":\"ScheduleAnyway\"},{\"maxSkew\":1,\"topologyKey\":\"kubernetes.io/hostname\",\"whenUnsatisfiable\":\"ScheduleAnyway\"}],\"ttlSecondsAfterEmpty\":120,\"ttlSecondsUntilExpired\":2592000}}\n"
        }
    },
    "spec": {
        "limits": {
            "resources": {
                "cpu": 1000
            }
        },
        "provider": {
            "subnetSelector": {
                "aws-ids": [
                    "subnet-XXXXX",
                    "subnet-XXXXX",
                    "subnet-XXXXX",
                    "subnet-XXXXX",
                    "subnet-XXXXX"
                ]
            }
        },
        "topologySpreadConstraints": [
            {
                "maxSkew": 1,
                "topologyKey": "topology.kubernetes.io/zone",
                "whenUnsatisfiable": "ScheduleAnyway"
            },
            {
                "maxSkew": 1,
                "topologyKey": "kubernetes.io/hostname",
                "whenUnsatisfiable": "ScheduleAnyway"
            }
        ]
    }
}
│ to:
│ Resource: "karpenter.sh/v1alpha5, Resource=provisioners", GroupVersionKind: "karpenter.sh/v1alpha5, Kind=Provisioner"
│ Name: "default", Namespace: ""
│ for: "/var/folders/0l/wk5qj7wj6sxf8pkkx6phm3y80000gn/T/847115903kubectl_manifest.yaml": admission webhook "validation.webhook.provisioners.karpenter.sh" denied the request: validation failed: v1alpha1.AWS.SubnetSelector: ReadString: expects " or n, but found [, error found in #10 byte of ...|aws-ids": [
    "subnet-XXXXX|..., bigger context ...|-XXXX"
},
"subnetSelector": {
    "aws-ids": [
        "subnet-XXXXX","subnet-XXXX|...:
│
│   with module.base_system.module.karpenter[
            0
        ].kubectl_manifest.karpenter_provisioner,
│   on /Users/fernando/modules/karpenter/karpenter.tf line 112, in resource "kubectl_manifest" "karpenter_provisioner":
│  112: resource "kubectl_manifest" "karpenter_provisioner" {

Resource Specs and Logs

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  requirements:
    - key: "topology.kubernetes.io/zone"
      operator: In
      values: ${azs}
    - key: "karpenter.sh/capacity-type"
      operator: In
      values: ${capacity-type}
    - key: "node.kubernetes.io/instance-type" #If not included, all instance types are considered
      operator: In
      values: ${instance-type}
    - key: "kubernetes.io/arch" #If not included, all architectures are considered
      operator: In
      values: ${arch}
  limits:
    resources:
      cpu: ${limits_resources_cpu}
  provider:
    instanceProfile: ${iam-instance-profile-id}
    subnetSelector:
      aws-ids: ${private_subnets}
    securityGroupSelector:
      aws-ids: ${security_groups}
    tags: ${tags}
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: "topology.kubernetes.io/zone"
      whenUnsatisfiable: ScheduleAnyway
    - maxSkew: 1
      topologyKey: "kubernetes.io/hostname"
      whenUnsatisfiable: ScheduleAnyway
  ttlSecondsAfterEmpty: ${ttlSecondsAfterEmpty}
  ttlSecondsUntilExpired: ${ttlSecondsUntilExpired}

# Deploying default provisioner for Karpenter autoscaler
data "kubectl_path_documents" "karpenter_provisioners" {
  pattern = "${path.module}/provisioners/default_provisioner.yaml"
  vars = {
    azs                     = jsonencode(var.aws_availability_zones)
    capacity-type           = jsonencode(var.capacity-type)
    instance-type           = jsonencode(var.instance-type)
    arch                    = jsonencode(var.arch)
    tags                    = jsonencode(data.aws_default_tags.aws_default_tags.tags)
    private_subnets         = join(", ", var.private_subnets)
    security_groups         = join(", ", var.security_groups)
    limits_resources_cpu    = var.limits_resources_cpu
    iam-instance-profile-id = aws_iam_instance_profile.karpenter.name
    eks-cluster-id          = var.eks_cluster_id
    ttlSecondsAfterEmpty    = var.ttlSecondsAfterEmpty
    ttlSecondsUntilExpired  = var.ttlSecondsUntilExpired
  }
}

[tzneal]

This is related to #1327 , we couldn't make that change now as it's backwards incompatible and would break existing customers.

[FernandoMiguel]

This is related to #1327 , we couldn't make that change now as it's backwards incompatible and would break existing customers.

Thanks for linking it.