Declarative way to add/delete nodegroups

[mumoshu]

TL;DR;

I want eksctl apply -f cluster.yaml for creating a cluster and creating/deleting nodegroups.

Updates to clusters and nodegroups is out of scope of this feature request.

Why do you want this feature?

While managing Kubernetes clusters, I believe it is common to just add nodegroup(s) to an existing cluster, to start serving different workloads on different infrastructure(instance type, volume size, host OS, kernel parameters, etc).

Kubernetes, by its nature, allows doing this without recreating the whole cluster. And I believe it is preferable to just add nodegroups rather than recreating the whole cluster every time you need to add nodegroups, because it takes just fewer time and financial cost.

I also prefer managing my cluster configuration as a whole in code, including Kubernetes control-planes provided by EKS and nodegroups provided by eksctl.

eksctl as of today has an API object for clusters that is called `ClusterConfig. It has dedicated field for nodegroup, which can be used for creating zero or more nodegroups along with the control-plane.

apiVersion: eksctl.io/v1alpha3
kind: ClusterConfig

metadata:
  name: cluster-1
  region: eu-north-1

nodeGroups:
  - name: ng-1
    instanceType: m5.large
    desiredCapacity: 10

However, there's no way to add nodegroups declaratively because:

• There's no way to nodegroups independently and declaratively, e.g. eksctl create nodegroup --config-file nodegroup.yaml, atm.
• There's no way to update nodeGroupsinside a ClusterConfig object after the initial creation.

Also note that, I'm unsure if the former actually works.

If something like eksctl cluster apply -f cluster.yaml is implemented to allow adding/removing embedded nodegroups, what happens when you run eksctl create nodegroup -f ng.yaml and then eksctl apply cluster -f cluster.yaml? Should the latter update the nodegroup created by the former command if the nodegroup names were equivalent? Then, which would be said to be the desired state of the nodegroup, ng.yaml or a nodegroup embedded in cluster.yaml?

What feature/behavior/change do you want?

I'd like to add eksctl apply [-f|--config-file] cluster.yaml, which supports creating a cluster and creating/deleting nodegroups only. Updates to clusters and nodegroups is out of scope of this feature request.

To me it isn't necessary to support updating every possible field in ClusterConfig. At least, initially, only addition/deletion of items under nodeGroups would be enough for this specific use-case.

The long-term goal would be to add support for updating more fields.

Personally, I prefer adding support to update for fields "one by one" with enough testing.

In other words, I prefer NOT to just render new CloudFormation stack templates from the cluster.yaml and then throwing away everything to CloudFormation for updates. It would introduce more failure cases than we can imagine, making the user support difficult. It would also leak too many implementation details like update errors from CloudFormation to the user, making the u/x bad.

[mumoshu]

Updates to nodegroups can be supported after #443 in the future. But again, it is out of the scope of this issue.

[mumoshu]

Please let me note that this feature isn't exclusive to eksctl create nodegroup -f ng.yaml + eksctl delete nodegroup.

#369 and eksctl create nodegroup -f should just be a primitive of eksctl that can be used to interact with nodegroups only. It can be used as similar as we can use kubectl create -f pod.yaml without dealing with Kubernetes Deployment resources.

A concern would be what I've wondered above:

If something like eksctl cluster apply -f cluster.yaml is implemented to allow adding/removing embedded nodegroups, what happens when you run eksctl create nodegroup -f ng.yaml and then eksctl apply cluster -f cluster.yaml? Should the latter update the nodegroup created by the former command if the nodegroup names were equivalent? Then, which would be said to be the desired state of the nodegroup, ng.yaml or a nodegroup embedded in cluster.yaml?

But this seems like easily addressed by introducing a kind of "owner reference", perhaps in a form of a stack tag. Node groups that are owned by the cluster but not in the desired state would be deleted. Node groups without owner refs to the cluster are never affected by updates of a cluster.

[errordeveloper]

@mumoshu to me it seems like we have this done now, I'm tempted to close it. If understand correctly, you would like to see eksctl <verb> --config-file=<path>, which is indeed a wanted feature, and it's mentioned in #19. I'm hoping to expand #19 into concrete issues sometime very soon, so we can start tackling those. Do you mind closing this?

[neelakansha85]

Just wanted to check if there is another open issue referencing the issue about updating current nodegroups config without having to delete and recreate cluster or even replace the nodegroups as highlighted in #443?
There could be changes to the nodegroup config as simple as related to the auto scaling group such as minSize, maxSize or labels which wouldn't need replacing the instances.

[michaelbeaumont]

Related #984

[bcbrockway]

Hey, tickets such as #2038 are being closed and linked to this one, but @mumoshu specifically mentions that updates to clusters with the config file are not in scope for this issue.

Can you please confirm if being able to make changes to the config file, apply them and have the relevant changes calculated and pushed a la terraform is in the pipeline for the future? We're currently looking to make the move from GCP to AWS and this is kind of a deal-breaker for us since currently the cluster config files in our repo may not necessarily match how the cluster is configured.

Without this we'll probably end up using terraform or forgoing EKS altogether in favour of kops.

[michaelbeaumont]

Hi @bcbrockway, you're right this may not be the best issue to use as a catch all, because of its history, but those features are indeed the intention for eksctl apply. It's in the pipeline for the future but unfortunately I don't have a timeframe for you at the moment

[michaelbeaumont]

Closing in favor of #2774