How to install/manage EKS add-ons for

[acim]

Is your request related to a new offering from AWS?

Yes. Since version 1.18 it is possible to install cluster add-ons to update vpc-cni, coredns and kube-proxy automatically. I don't see this is possible with this module at the moment or I am missing something.

https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html#adding-vpc-cni-eks-add-on
https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html#adding-coredns-eks-add-on
https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html#adding-kube-proxy-eks-add-on

Is your request related to a problem? Please describe.

After updating to the new cluster version it is necessary to update vpc-cni, coredns and kube-proxy manually. This was the case up to version 1.17, but with version 1.18 there seems to be a better way described at the links above. Can this module install/manage these add-ons?

Describe the solution you'd like.

Manage add-ons within this module.

Describe alternatives you've considered.

I can still update these manually, but it is time consuming.

https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html

[daroga0002]

I assume you can use new resource in terraform:
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_addon

in module this is not possible at the moment. It was added to AWS provider in 3.40.0 release (May 13, 2021).

Looking also that this has some restrictions from AWS side:

Amazon EKS add-on can only be used with Amazon EKS Clusters running version 1.18 with platform version eks.3 or later because add-ons rely on the Server-side Apply Kubernetes feature, which is only available in Kubernetes 1.18 and later.

and module suppose to support also earlier versions so this require some special handling.

Most probably you can just add to your code:

resource "aws_eks_addon" "vpc-cni" {
  cluster_name = module.eks_cluster.cluster_id
  addon_name   = "vpc-cni"
}

[jaimehrubiks]

Has anyone tried that? Please share your experience :)

[barryib]

Here is how I handle this in my code

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "16.2.0"
  # insert the 8 required variables here
}

resource "aws_eks_addon" "vpc_cni" {
  count = var.addon_create_vpc_cni ? 1 : 0

  cluster_name      = module.eks.cluster_id
  addon_name        = "vpc-cni"
  resolve_conflicts = "OVERWRITE"
  addon_version     = var.addon_vpc_cni_version

  tags = local.tags
}

resource "aws_eks_addon" "kube_proxy" {
  count = var.addon_create_kube_proxy ? 1 : 0

  cluster_name      = module.eks.cluster_id
  addon_name        = "kube-proxy"
  resolve_conflicts = "OVERWRITE"
  addon_version     = var.addon_kube_proxy_version

  tags = local.tags
}

resource "aws_eks_addon" "coredns" {
  count = var.addon_create_coredns ? 1 : 0

  cluster_name      = module.eks.cluster_id
  addon_name        = "coredns"
  resolve_conflicts = "OVERWRITE"
  addon_version     = var.addon_coredns_version

  tags = local.tags
}

For right add-on versions, you can use aws eks describe-addon-versions. So far, it works like a charm.

[tnimni]

Hello,

@daroga0002
Why is it not possible in module?

thank you

[daroga0002]

Hello,

@daroga0002

Why is it not possible in module?

thank you

Because this code is not implemented into module. Look solution above from @barryib how to add it.

[tnimni]

Hello,
@daroga0002
Why is it not possible in module?
thank you

Because this code is not implemented into module. Look solution above from @barryib how to add it.

@daroga0002
Hi,
Thank you, I taught you meant it is not possible to add to module

I'll open a PR once I have some time

[ashtonian]

any way to enable external-snat for the cni plugin with the workaround ?

[jaimehrubiks]

Not sure about external-snat on the cni. But I can share my experience with coredns, and it is bad sadly. Using the eks addon for coredns there is no way to add something like tolerations. I patch the resource, but every now and then the deployment is changed back.

[tnimni]

I opened pr #1443 to allow deployment of those addons via the module

[tnimni]

@ashtonian for which secnario you want to enable external snat?

[ashtonian]

@tnimni I may be wrong but I am under the impression if the pod needs to be directly interacted with from the internet I need external snat, for something like hosting a clustered mqtt broker for devices to connect directly to.

[tnimni]

@ashtonian Hi,
Enabling or disabling snat is done via the env param on the pod, father the update all aws-node pods to restart with the new env params.
if you want to have direct access from the internet meaning pod communicate directly to the internet gateway, it require snat enabled, AWS_VPC_K8S_CNI_EXTERNALSNAT=false which is the default.
meaning you don't need to set anything
if you want snat disabled meaning traffic routed via nat gateway you want the opposite.

I admit the terms are confusing. this is why I asked what is your scenrio.

[ashtonian]

@tnimni Thanks! that does clear it up.

[gavinclarkeuk]

Just on the external-snat issue - we have a use case where we need to enable this envvar so our EKS cluster can communicate with other private VPCs inside our AWS accounts via transit gw's. I understand we can set the envvar via kubctl, but it would be great to be able to set that in terraform via this module. Is that possible?

[DaemonDude23]

@gavinclarkeuk Not that I'm aware of, as the underlying resource doesn't look to have configuration options for environment variables.
If I were you go with AWS's VPC CNI Helm Chart or something along those lines. I recently switched to using EKS Add-ons but for future flexibility purposes, I'm having second thoughts.

[skakanigit]

Just on the external-snat issue - we have a use case where we need to enable this envvar so our EKS cluster can communicate with other private VPCs inside our AWS accounts via transit gw's. I understand we can set the envvar via kubctl, but it would be great to be able to set that in terraform via this module. Is that possible?

@gavinclarkeuk / @ashtonian : I hope you might have already found the solution to enable/disable external-snat through terraforming. But in case if you are still wondering here is how we did it when we ran into a similar issue.

resource "null_resource" "generate_kube_session" {
  triggers   = {
    always_run = timestamp()
  }
  provisioner "local-exec" {
    command = "aws eks --region ${var.consul_region} update-kubeconfig --name ${module.eks.eks_cluster_name}"
  }
  depends_on = [
    module.eks]
}

resource "null_resource" "enable_external_snat_rule" {
  triggers   = {
    always_run = timestamp()
  }
  provisioner "local-exec" {
    command = "kubectl set env daemonset -n kube-system aws-node AWS_VPC_K8S_CNI_EXTERNALSNAT=true"
  }
  depends_on = [
    null_resource.generate_kube_session]
}

[sogajeffrey]

Is there any update on this? I was able to add EKS addons separately from the module but theres a race condition when creating or destroying the entire cluster. Its MUCH cleaner if implemented in module with proper depends_on mappings..

[tho93]

Is there a Terraform data source to retrieve the latest addon versions? Basically what we get from the aws cli:
aws eks describe-addon-versions --addon-name kube-proxy --query 'addons[*].addonVersions[0].addonVersion'

[antonbabenko]

This issue has been resolved in version 18.0.0 🎉

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.