Controller Support for Worker Node Taints

[danbudris]

The EKS-A Controller should reconcile modifications to the taints field of the workerNodeGroupConfigurations in such a way that new nodes are rolled out and tainted appropriately.

In order to provide controller support for taints, we'll need to add a mechanism by which we generate new kubeadmconfigtemplates with an update joinConfiguration containing the requisite taints, and then modify the machineDeployemnts which reference those kubeadmconfigtemplates to reference the new kubeadmconfigtemplate; this way we can trigger a rollout of the machinedeployments, causing new machines to be launched, and the updated joinConfiguration in the kubeadmconfigtemplate to go into effect. This will also address SSH key rolling.

As of right now, upstream intentionally does not support 'immutable' kubeadmconfigtemplates, but the official guidance is to treat them in the same way we treat the immutable machineTemplates -- e.g. create a new one with the updates, modify the machineDeployment to use it, trigger rollout & node recreation.

See:

• Make KubeadmConfig and the KubeadmConfigTemplate immutable kubernetes-sigs/cluster-api#4910
• https://cluster-api.sigs.k8s.io/tasks/updating-machine-templates.html

Originally posted by @danbudris in #189 (comment)

[danbudris]

support for reconciliation of taints we merged in with #1175