EKS: Bootstrap container for calculating K8s max-pods value for different CNI configurations in a self-managed nodegroup

[etungsten]

Originally Bottlerocket hosts determined the Kubernetes max pods value by mapping the host’s instance type to the number of ENIs available. We did this by checking EKS’s eni-max-pods file during boot up.

However, the maximum number of pods supported by an EC2 node now depends on several factors in addition to the number of ENIs an instance type has. These new factors cannot be determined by the Bottlerocket host at runtime:

• The deployed amazon-vpc-cni plugin version
• Whether CNI custom networking is enabled
• Whether prefix delegation is enabled (amazon-vpc-cni >=v1.9.0 required)

EKS provides a helper script for users to manually calculate the number of maximum pods their nodes support before launching said nodes. Below we demonstrate how we can set up a Bottlerocket bootstrap container for calculating and configuring this max-pods value in a slightly more automated fashion.

Here is the Dockerfile for the example bootstrap container image:

FROM alpine
ARG MAX_PODS_CALC_SCRIPT_URL=https://raw.githubusercontent.com/awslabs/amazon-eks-ami/v20210722/files/max-pods-calculator.sh
RUN apk --update add curl bash jq
# Install aws-cli
RUN apk add --no-cache \
        python3 \
        py3-pip \
    && pip3 install --upgrade pip \
    && pip3 install \
        awscli \
    && rm -rf /var/cache/apk/*
# Downloads the EKS helper script
RUN curl -o /usr/local/bin/max-pods-calculator.sh $MAX_PODS_CALC_SCRIPT_URL \
    && chmod +x /usr/local/bin/max-pods-calculator.sh
ADD bootstrap-script ./
RUN chmod +x ./bootstrap-script
ENTRYPOINT ["./bootstrap-script"]

The bootstrap-script script wraps around the max-pods-calculator script and is structured like so:

#!/usr/bin/env bash

# Source environment variables defined in bootstrap container's user-data
USER_DATA_DIR=/.bottlerocket/bootstrap-containers/current
source "$USER_DATA_DIR/user-data"

if [ "${CUSTOM_NETWORKING}" = "true" ]; then
  CUSTOM_NETWORKING_ARG="--cni-custom-networking-enabled"
fi
if [ "${PREFIX_DELEGATION}" = "true" ]; then
  PREFIX_DELEGATION_ARG="--cni-prefix-delegation-enabled"
fi

# Runs the max-pods-calculator script to calculate max pods
max_pods=$(max-pods-calculator.sh \
  --instance-type-from-imds \
  --cni-version "${CNI_VERSION}" \
  ${CUSTOM_NETWORKING_ARG} \
  ${PREFIX_DELEGATION_ARG} \
  ${CNI_MAX_ENI:+--cni-max-eni "${CNI_MAX_ENI}"})
if [ ${?} -ne 0 ]; then
  echo "ERROR: Failed to calculate max-pods value using the max-pods-calculator helper script: ${max_pods}" >&2
  exit 1
fi

# Set the max-pods setting via Bottlerocket's API
apiclient set kubernetes.max-pods=${max_pods}

if [ ${?} -ne 0 ]; then
  echo "ERROR: Failed set kubernetes.max-pods setting via Bottlerocket API" >&2
  exit 1
fi

The script depends on several environment variables to calculate the max-pods value. They will need to be set via the bootstrap-containers user-data.

user_data="$(cat <<EOF
export CNI_VERSION=<aws-vpc-cni plugin version, e.g. 1.9.0>
export CUSTOM_NETWORKING=true
export PREFIX_DELEGATION=true
# optional, caps the number of ENIs allocated to pods
export CNI_MAX_ENI=10
EOF
)"
# Base64 encode the bootstrap container's user-data
encoded_user_data=$(echo "${user_data}" | base64 -w0)

Now we can add the settings for the bootstrap container in Bottlerocket’s instance user data:

[settings.bootstrap-containers.max-pods-calculator]
source = "<URL for the bootstrap container>"
essential = false
mode = "always"
# Set the encoded user-data here
user-data = "${encoded_user_data}"

Any Bottlerocket hosts launched with the instance user-data above should automatically configure settings.kubernetes.max-pods value according to the CNI configuration values you’ve set in the bootstrap container’s userdata.

[James-Quigley]

This seems to still work. However I will note that setting CNI_MAX_ENI is not optional, as the first check in the bootstrap script above exits if it is unset.

Additionally in the example userdata, it should be CUSTOM_NETWORKING instead of CUSTOM_NETWORKIN

[etungsten]

Thanks for trying this out! Glad to hear that it worked. I updated the instructions based on your feedback. I removed the check for CNI_MAX_ENI and fixed the typo in CUSTOM_NETWORKING