feat: Update serverless clusters running EKS Farage to maximize resource utilization on CoreDNS

[bryantbiggs]

What does this PR do?

• Update serverless clusters running EKS Farage to maximize resource utilization on CoreDNS

Motivation

• When running on EKS Fargate, pods are assigned Fargate task sizes that meet the resource request/limits specified. If a user requests values that are much lower than the closest task size, these resources will go to waste. Instead, users should calculate the request/limits they require, find the closest Fargate task size, and then increase their requests/limits to fit within that next closest task size to maximize resources that are provisioned. You can see the Fargate resource graduations here https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html
• Resolves Fargate-serverless example launches CoreDNS pods with request/limits below the minimum Fargate values #1327

More

• Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
• Yes, I have added a new example under examples to support my PR
• Yes, I have created another PR for add-ons under add-ons repo (if applicable)
• Yes, I have updated the docs for this feature
• Yes, I ran pre-commit run -a with this PR

Note: Not all the PRs require a new example and/or doc page. In general:

• Use an existing example when possible to demonstrate a new addons usage
• A new docs page under docs/add-ons/* is required for new a new addon

For Moderators

• E2E Test successfully complete before merge?

Additional Notes

[csantanapr]

I think is good to include this other link https://docs.aws.amazon.com/eks/latest/userguide/fargate-pod-configuration.html
Don't know if you want to add this link or replace the current one with this one.

This one shows that 256MB are being reserved for kubelet, kube-proxy, containerd

Fargate adds 256 MB to each pod's memory reservation for the required Kubernetes components (kubelet, kube-proxy, and containerd).
Fargate rounds up to the following compute configuration that most closely matches the sum of vCPU and memory requests in order to ensure pods always have the resources that they need to run.

And also provides an example

The additional memory reserved for the Kubernetes components can cause a Fargate task with more vCPUs than requested to be provisioned. For example, a request for 1 vCPU and 8 GB memory will have 256 MB added to its memory request, and will provision a Fargate task with 2 vCPUs and 9 GB memory, since no task with 1 vCPU and 9 GB memory is available.

[csantanapr]

Just thinking out loud, I think having this as tip and script/tool to scanned a cluster using fargate and calculate total waste of resources. This can be a cost and performance optimization tip/script during evaluation of a cluster config.

[csantanapr]

/lgtm

[bryantbiggs]

looks good now

Labels:               eks.amazonaws.com/component=coredns
                      eks.amazonaws.com/fargate-profile=kube-system-2a
                      k8s-app=kube-dns
                      pod-template-hash=6f984dc4dc
Annotations:          CapacityProvisioned: 0.25vCPU 0.5GB # <= Smallest capacity provisioned
                      Logging: LoggingDisabled: LOGGING_CONFIGMAP_NOT_FOUND
                      eks.amazonaws.com/compute-type: Fargate
                      kubernetes.io/psp: eks.privileged
Status:               Running
IP:                   10.0.10.100
IPs:
  IP:           10.0.10.100
Controlled By:  ReplicaSet/coredns-6f984dc4dc
Containers:
  coredns:
    Container ID:  containerd://51755a6101eef18147e8a529ee0f84996f4d227bdec917277c659dd4c4b576a2
    Image:         602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.8.7-eksbuild.3
    Image ID:      602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns@sha256:871a47b5773bbde084c8e9f12e4321a168ddd578521ffae8b55b3dd982ddc66a
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Fri, 13 Jan 2023 10:22:55 -0500
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     250m # <= Using all of available CPU for smallest capacity
      memory:  256M # <= Using all of the available memory for the smallest capacity *AFTER account for overhead
    Requests:
      cpu:     250m # <= Using all of available CPU for smallest capacity
      memory:  256M # <= Using all of the available memory for the smallest capacity *AFTER account for overhead