Add a switch to scope node groups to a single subnet

[avestuk]

Is your request related to a new offering from AWS?

No

Is your request related to a problem? Please describe.

Currently node groups are created with all the subnets that are passed in via subnet_ids. This causes issues when using Cluster Autoscaler and running stateful workloads backed by EBS volumes, as the EBS volumes are AZ locked. Having node group scoped to a single subnet is also recommended by AWS when working with stateful workloads.

The module currently does check to see if each value in the eks_managed_node_groups has a subnet_id field but it's somewhat difficult to generate the correct data structure downstream of the module.

The required data structure for creating 3 node groups that only have a single subnet associated.

managed_node_groups  = {
    t3medium-a = {
        instance_types = [
            "t3.medium",
            "t3a.medium",
         ]
        labels         = {
            instance_family = "t3x.medium"
            type            = "spot"
         }
        subnet_ids     = [
            "a",
         ]
     }
    t3medium-b = {
        instance_types = [
            "t3.medium",
            "t3a.medium",
         ]
        labels         = {
            instance_family = "t3x.medium"
            type            = "spot"
         }
        subnet_ids     = [
            "b",
         ]
     }
    t3medium-c = {
        instance_types = [
            "t3.medium",
            "t3a.medium",
         ]
        labels         = {
            instance_family = "t3x.medium"
            type            = "spot"
         }
        subnet_ids     = [
            "c",
         ]
     }
 }

Describe the solution you'd like.

A variable bool to generate managed node groups per subnet passed in via subnet_ids. So for example if we pass in a single managed node group t3medium and three subnet_ids we'd end up with the three t3medium node groups above.

Currently we are achieving this downstream using the following:

  eks_managed_node_groups = merge([
    for subnet_id in local.subnet_ids : {
      for node_group_name, node_group_config in var.eks_managed_node_groups :
      format("%s-%s", node_group_name, subnet_id) => {
        instance_types = node_group_config.instance_types
        labels         = node_group_config.labels
        subnet_ids     = [subnet_id]
      }
    }
  ]...)

Describe alternatives you've considered.

Documenting the solution to combine the subnet_ids and the eks_managed_node_groups so people can use that approach if they wish to do this.

Additional context

[bryantbiggs]

unfortunately I don't think this is feasible from the module's perspective. there are two options users can take:

1. The long form like you've shown above where subnet_ids is a list containing one subnet ID
2. Users can attempt to construct their own map comprehension to achieve this thought I suspect you could face issues with Partial/Progressive Configuration Changes hashicorp/terraform#4149

[bryantbiggs]

also - I would look into https://karpenter.sh/ instead of cluster autoscaler since it handles this scenario (still early days for that project though)

[avestuk]

@bryantbiggs Could you expand on why you don't think this is feasible from the module perspective? We're currently creating the necessary inputs upstream so I don't understand why this couldn't be handled by the module. Although if it's not desirable for the module to add the additional logic I can understand that.

I'll definitely have a look at Karpenter though thanks for the suggestion!

[bryantbiggs]

yes, the module is designed to afford users the opportunity to do this looping/comprehension to build their desired node groups, but that is because the module was designed to create "one of" not a defined set at a time. trying to take in a list and build out the desired node groups would severely impact the flexibility that is currently provided to users. therefore, its just not something that we can consider at this time

[avestuk]

Ok, thanks for taking the time to explain.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.