Support for different availability zones per-worker

[chancez]

I have issues

I'd like to configure a separate ASG per zone for better interaction with statefulsets and PVCs.

I'm submitting a...

• feature request
• support request - read the FAQ first!

What is the current behavior?

I believe this may be possible by configuring subnets for each worker object in the worker_groups list since this maps those subnets into the vpc_zone_identifier. However this is less than ideal because I'd prefer to simply just the simpler AZ names.

What's the expected behavior?

Ability to configure AZ per worker like so:

module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "8.1.0"
....
subnets = concat(var.networking.pub_subnet, var.networking.private_subnet)
worker_groups = [
{
instance_type = "m5.xlarge"
asg_min_size = 2
asg_max_size = 2
asg_desired_capacity = 2
availability_zones = ["us-west-1a"]
},
{
instance_type = "m5.xlarge"
asg_min_size = 2
asg_max_size = 2
asg_desired_capacity = 2
availability_zones = ["us-west-1b"]
}
]
}

Are you able to fix this problem and submit a PR? Link here if you have already.

I have a WIP locally but it's unclear if I'll need to change how vpc_zone_identifier is set when this option is configured. Currently vpc_zone_identifier is set to the values of the subnets variable on the module, or the subnets variable of the worker_groups object if set. I know that both availability_zones and vpc_zone_identifier can both be set, but then you need to ensure the subnets used belong to the AZ, which is a bit complicated.

I'm currently thinking that if I was to add an availability_zones variable to workers_group_defaults_defaults and the autoscaling group resources, then when it's empty the behavior is the same as today. If non-empty then it will cause vpc_zone_identifier to be unset (null/empty list) and instead availability_zones will be set.

[max-rocket-internet]

Duplicate of and solutions are provided in #346

Let us know if they don't work but I think this is very elegant:

worker_groups = [
    for subnet in data.aws_subnet_ids.private.ids :
    {
      subnets              = [subnet],
      asg_desired_capacity = 3,
      instance_type        = "m5.large",
      ami_id               = data.aws_ami.eks_worker.id
    }
  ]

[chancez]

I already mentioned I explicitly don’t want to specify subnets as a method for setting AZs. Your simple example of looping also gets more complicated when you have multiple subnets per AZ like private/public.

This approach using subnets is difficult to use and doesn’t immediately make it obvious what’s being done. Unless you know what subnets you set in which AZ it’s very painful.

I would prefer a simpler method of just being able list the AZs by name.

[chancez]

If I opened a PR which accomplished what I’m asking for would you be open to integrating it? If it ends up too complicated i can concede to using subnets.

[max-rocket-internet]

I already mentioned I explicitly don’t want to specify subnets as a method for setting AZs

My apologies. I'll reopen the issue 🙏

[max-rocket-internet]

Unless you know what subnets you set in which AZ it’s very painful.

I would imagine the subnet creation is also done also in terraform so you should know the AZ? But if you subnets are not created in Terraform then I guess it could be a problem.

You could always just create your own nice/clean list of subnets per AZ outside of the module and then loop over that like I posted above. Or use the subnet data resource?

If I opened a PR which accomplished what I’m asking for would you be open to integrating it?

Hard to imagine what it looks like but yes of course, as long as:

• It's tested
• Doesn't break anything else
• The complexity it adds is worth the benefit for most users

[chancez]

For us, our subnets are defined in a separate terraform module/deployment. They're accessible, and it's relatively doable to map them to AZes but it's still fairly opaque unless you know what your looking at.

Using the subnet data resource is actually a decent idea to do it without changing the module, but I'll also look at how to setup a nicer list per AZ to see if I can make it more readable before trying to change the module.

[max-rocket-internet]

Using the subnet data resource is actually a decent idea to do it without changing the module

Sounds good. You could loop over a list of subnet IDs to create the data resources, then create a new list filtered based on AZ name. I wrote some iteration examples here: https://gist.github.com/max-rocket-internet/0b6955a80b0fab1b3d3fe81f8953c0a2

[chancez]

So far I've managed to make things work using just subnets, and with some refactoring it isn't too bad so I'd say this isn't a huge priority. I also realized I actually wanted to specify my subnets anyways because we'll have nodes in certain subnets which have access to some resources that others don't, so I think this isn't a big deal for us. I do think being able to just specify the AZs would be nice, but I realize for many people's uses, once you want to do that you may be specifying the subnets per ASG for other reasons anyways.

[max-rocket-internet]

OK cool. Glad you made it work!

Managing and creating subnets is outside of this module and I think any complexity there is on the user to solve before passing subnet IDs to this module. So I will close this issue now.

[pepgid]

Can you provide a working example, which alleviates the autoscaling storage incongruence issue?

[casey-robertson]

Can you provide a working example, which alleviates the autoscaling storage incongruence issue?

I’m currently figuring this out as well. We use Pulumi but it’s imported from Terraform so the same issue applies. Up until now we have deployed multi-AZ node groups but that’s not going to fly any more. We pass the private subnets in from our networking stack whole sale.

I will review the mentioned examples but would also encourage people to look into how eksctl handles this because they can do both - you can specify subnets or AZ(s) including a single one.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.