fix: Don't tag self managed node security group with kubernetes.io/cluster tag

[imdevin567]

Description

This removes the forced kubernetes.io/cluster tag in self managed node security groups in favor of the user supplying it themselves.

Motivation and Context

We currently tag the created node security group with the kubernetes.io/cluster/clustername = owned tag. We are also tagging the self managed node security group with the same tag, which causes a conflict launching load balancers:

Error syncing load balancer: failed to ensure load balancer: Multiple tagged security groups found for instance i-aaaaaab0ab17adb5d; ensure only the k8s security group is tagged;

Since we don't add this tag to the EKS managed node security group, this will ensure consistency among node groups.

Breaking Changes

No breaking change

How Has This Been Tested?

• I have tested and validated these changes using one or more of the provided examples/* projects

[bryantbiggs]

I think the alternative here would be to not let the self managed node group create a security group, and instead supply your own security group

[imdevin567]

That's fine, but right now the created security group for self managed nodes is unusable since it results in multiple tagged security groups. It can be worked around, but we should probably fix (or remove) the functionality that isn't working correctly.

[bryantbiggs]

@daroga0002 thoughts?

[daroga0002]

@imdevin567 could you pass here problematic module configuration which is solved by this PR?

[imdevin567]

Sure thing.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 18.0"

  cluster_version = "1.21"
  cluster_name    = "${var.env}-eks-cluster"
  vpc_id          = var.vpc_id
  subnet_ids      = concat(tolist(var.public_subnets), tolist(var.private_subnets))

  self_managed_node_group_defaults = {
    subnet_ids = var.private_subnets
  }

  self_managed_node_groups = {
    ci = {
      name                 = "${var.env}-ci"
      instance_types       = [var.ci_instance_type]
      min_size             = 1
      max_size             = 2
      desired_size         = 1
      bootstrap_extra_args = "--kubelet-extra-args -'-node-labels=mida.io/node_group=ci'"
    }
  }
}

This creates two problematic security groups: develop-eks-cluster-node20220112155729835300000001 which is the created node security group, and develop-ci-node-group-20220112202026375000000002 which is the created security group for the ci self managed node group. Both of these security groups contain the annotation:

kubernetes.io/cluster/develop-eks-cluster = owned

Both security groups are attached to the self managed nodes, which causes the following error when trying to create LoadBalacer resources:

Error syncing load balancer: failed to ensure load balancer: Multiple tagged security groups found for instance i-aaaaaab0ab17adb5d; ensure only the k8s security group is tagged;

[bryantbiggs]

please add back in the "Name" tag, then we should be good to go with this one

[imdevin567]

Should be all set now.

[bryantbiggs]

@antonbabenko good to go 👍🏽 - we might see an issue pop up since its removing a tag, but it is the right move to remove it and let user opt in to when and where they want to use that tag

[antonbabenko]

This PR is included in version 18.2.7 🎉

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.