(aws-eks): managed nodegroup for spot instances

[pahud]

Amazon EKS now supports provisioning and managing EC2 Spot Instances in managed node groups
https://aws.amazon.com/tw/blogs/containers/amazon-eks-now-supports-provisioning-and-managing-ec2-spot-instances-in-managed-node-groups/

eksctl just introduced the new --spot flag which seems to create a new launch template with spot options and pass this template to create the spot-only nodegroup.

I was wondering how to create similar experience with aws-eks. The Nodegroup L2 construct actually can accept the launch template as the construct property, however, users need to bake their own LT with spot options before they can pass it to NodegroupProps. It doesn't make sense to add a new spot property for NodegroupProps as this is actually a high level abstraction, but making people easily create spot nodegroup with CDK is really helpful.

Some options:

1. create aws-eks-patterns L3 and make SpotNodegroup a L3 construct
2. let's create a 3rd party construct lib like cdk-eks-spot-nodegroup

I'd vote option 1.

@iliapolo wdyt?

ref: https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[iliapolo]

@pahud I actually think this should be part of the core library, its definitely an important capability and we should provide an experience as easy as --spot.

I think we can start with enabling this via the current launchTemplateSpec option but provide an easy way to create a compatible launch template.

const spots = new eks.SpotLaunchTemplateSpec(...);
new eks.Nodegroup(this, 'SpotNodeGroup', {
  ...,
  launchTemplateSpec: spots,
})

When we introduce proper ec2.LaunchTemplate, this will become:

const spots = new eks.SpotLaunchTemplate(...); // will now extend `ec2.LaunchTemplate`.
new eks.Nodegroup(this, 'SpotNodeGroup', {
  ...,
  launchTemplate: spots,
})

Thoughts?

[pahud]

@iliapolo Yes. That will be easier. Please leave this PR for me. I'll do it in the next few days.

[iliapolo]

@pahud Let's have another API discussion here before you actually start coding? still feels a little awkward to me even though the approach is solid.

[abierbaum]

@iliapolo @pahud FWIW, it would be great if the high level api for the cluster just allowed something similar to:

cluster.addNodegroupCapacity('spot-node-group', {
  instanceTypes: [new ec2.InstanceType('m5.large'), new ec2.InstanceType('m5.xlarge')],
  minSize: 4,
  spot: true,
});

Interacting with launch templates seems like an advanced concept that isn't needed for the simple case of just adding more capacity.

[pahud]

@pahud Let's have another API discussion here before you actually start coding? still feels a little awkward to me even though the approach is solid.

Hi @iliapolo I have yet to start coding. Please let me know what's in your mind.

RE @abierbaum I believe there would be some scenarios for the EKS with Spot instances integration:

1. managed NG with CapacityType='spot'

CapacityTye is a new option in API/SDK, see Managed node group capacity types, however, it's not yet available in cloudformation(AWS::EKS::Nodegroup) at this moment. When we specify CapacityType='spot', behind the scene an AutoScaling Group with launch template of single instance type across all AZs will be provisoned.

2. managed NG with `CapacityType='on-demand'

Same as above. But only on-demand single instance type.

3. managed NG with multiple instance types and purchase options

According to Considerations for selecting a capacity type:

When deploying your node group with the Spot capacity type that's using a custom launch template, use the API to pass multiple instance types instead of passing a single instance type through the launch template. For more information about deploying a node group using a launch template, see Launch template support.

To achieve this, we'll need to bake the launch template for the nodegroup, which would be a little bit challenging. Good to hide it behind the construct.

Given this, I think we probably can design the Nodegroup API like this

cluster.addNodeGroup(scope, id, {
   capacityType:  CapacityType.SPOT,
   instanceTypes: InstanceType[],
})

or simply

cluster.addNodeGroup(scope, id, {
   spot: true,
   instanceTypes: InstanceType[],
})

And we create the nodegroup and bake the LT with custom resource. WDYT?

[pahud]

this is weird capacityType is mentioned in cfn nodegroup document but actually not available

If you specify Spot for capacityType, then we recommend specifying multiple values for instanceTypes. For more information, see Managed node group capacity types and Launch template support in the Amazon EKS User Guide.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-eks-nodegroup.html

If cfn can support this, that would be easier for implementation in cdk.
aws-cloudformation/cloudformation-coverage-roadmap#712

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.