Proper setup procedure of eniConfig?

[yuanlinios]

Customized security groups and subnets are required for my pods, so I need the eniConfig as it is documented here https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html

From the PR #165, I realize the eni config only applies to new ENI allocated after worker node is annotated.

The problem is: after I join worker nodes to EKS cluster, and before I apply the eniConfig, the IPAMD in worker nodes already allocated ENI and ip addresses. So even I annotate worker node properly, before a new ENI is allocated, all my pods created with IP addresses already allocated will still be placed in wrong subnets.

Is there any proper procedure to avoid this problem?

[ewbankkit]

@yuanlinios If you start the CNI pod with AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true then pods running without hostNetwork: true will not get assigned IP addresses (and therefore not be scheduled) until nodes are annotated with k8s.amazonaws.com/eniConfig values. You will either have to annotate the nodes manually or write a controller (that runs with with hostNetwork: true) that annotates the nodes programatically.

[yuanlinios]

Hello @ewbankkit

Thanks a lot for the explanation! It makes sense to me