eks-prow-build-cluster: Use dedicated Managed Node Groups (MNGs) per Availability Zone (AZ)

[xmudrii]

xref #6303 (see for more details and context)

As per recommendations received from the AWS support and @tzneal, we're replacing blue and green node groups with node groups per AZ. In other words, old node groups had instances in all three AZs. Now, we have a dedicated node group for each AZ. This is a short-term solution to fix the stability issues that we're facing.

This has been successfully rolled out to canary, I'll do prod rollout once this PR is reviewed and merged.

Notes:

• We plan to switch to Karpenter long-term (eks-prow-build-cluster: Use Karpenter instead of cluster-autoscaler #5168)
• We didn't want to suspend AZRebalacing process because it's not natively supported by EKS, so we opted in for dedicated node groups as that solution can be fully-automated using Terraform
• We have dedicated Terraform objects and variables for each Node Group. I explicitly didn't want to use count because that would make rolling out upgrades too complicated

Follow-ups:

• Remove blue/green Terraform variables and objects
• Update docs to reflect changes in the upgrade procedure

/assign @upodroid @ameukam @dims

[upodroid]

LGTM

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: upodroid, xmudrii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• infra/aws/terraform/prow-build-cluster/OWNERS [upodroid,xmudrii]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[upodroid]

cancel the hold when you are ready to merge
/hold

[xmudrii]

Let's get this merged and I'll create a new PR for follow-ups
/hold cancel

[bryantbiggs]

FYI - you can use eks_managed_node_group_defaults to set the default values you wish to use across all of the managed nodegroups, and then any specific settings unique to that nodegroup can be set in the nodegroup definition. You can also set the defaults and still override the value in the nodegroup definition as well

So roughly speaking, something like the following might help cut down on the copy+pasta across nodegroup definitions:

  eks_managed_node_group_defaults = {
    use_name_prefix = true
    
    taints = var.node_taints_build
    labels = var.node_labels_build
    ... anything else that you want common across the nodegroups
  }
  
  

[xmudrii]

That's neat, thanks for pointing it out! I'll keep it in mind and see if we can make use of it the next time we do some rollout