Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add tenancy option dedicated on LaunchTemplate #6360

Closed
wants to merge 5 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -433,6 +433,20 @@ spec:
rule: self.all(k, k !='karpenter.sh/nodeclaim')
- message: tag contains a restricted tag matching karpenter.k8s.aws/ec2nodeclass
rule: self.all(k, k !='karpenter.k8s.aws/ec2nodeclass')
tenancy:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are your thoughts about doing something smarter here? Can Karpenter be aware of your dedicated instances and just use them? What happens if you run out of dedicated instances for the EC2NodeClass to launch?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is placement.tenancy information in the response of ec2 launch instance.
https://docs.aws.amazon.com/ko_kr/AWSEC2/latest/APIReference/API_Placement.html

Using this, We can enter tenancy information in nodeclass status, but I don't know if any other method can be implemented other than logging that the dedicated instance of the currently requested Instance requirement is Insufficient.

description: |-
Tenancy of the instance. An instance with a tenancy of dedicated runs on single-tenant hardware.


If not set, Tenancy will be set to default.
Currently, Karpenter only support default and dedicated option.
For more information,
See the AWS::EC2::LaunchTemplate Placement Document.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-launchtemplate-placement.html
enum:
- default
- dedicated
type: string
userData:
description: |-
UserData to be applied to the provisioned nodes.
Expand Down
10 changes: 10 additions & 0 deletions pkg/apis/v1beta1/ec2nodeclass.go
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,16 @@ type EC2NodeClassSpec struct {
// +kubebuilder:default={"httpEndpoint":"enabled","httpProtocolIPv6":"disabled","httpPutResponseHopLimit":2,"httpTokens":"required"}
// +optional
MetadataOptions *MetadataOptions `json:"metadataOptions,omitempty"`
// Tenancy of the instance. An instance with a tenancy of dedicated runs on single-tenant hardware.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you able to workaround this limitation by having the VPC tenancy be dedicated? Or do you need a mix of dedicated and non-dedicated instances in the VPC?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typically, dedicated tenancy has less available instance capacity in a region than default tenancy. This is a case where default and tenancy was mixed and used within a VPC.

//
// If not set, Tenancy will be set to default.
// Currently, Karpenter only support default and dedicated option.
// For more information,
// See the AWS::EC2::LaunchTemplate Placement Document.
// https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-launchtemplate-placement.html
// +kubebuilder:validation:Enum:={default,dedicated}
// +optional
Tenancy *string `json:"tenancy,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we even need default or can we just assume that not specifying this option assumes that you are using "shared"?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If not specified, tenancy is "default (shared)".
If wish, We can also specify tenancy: "default" in yaml.

// Context is a Reserved field in EC2 APIs
// https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateFleet.html
// +optional
Expand Down
9 changes: 9 additions & 0 deletions pkg/apis/v1beta1/ec2nodeclass_validation.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ const (
amiFamilyPath = "amiFamily"
tagsPath = "tags"
metadataOptionsPath = "metadataOptions"
tenancyPath = "tenancy"
blockDeviceMappingsPath = "blockDeviceMappings"
rolePath = "role"
instanceProfilePath = "instanceProfile"
Expand Down Expand Up @@ -79,6 +80,7 @@ func (in *EC2NodeClassSpec) validate(_ context.Context) (errs *apis.FieldError)
in.validateAMISelectorTerms().ViaField(amiSelectorTermsPath),
in.validateMetadataOptions().ViaField(metadataOptionsPath),
in.validateAMIFamily().ViaField(amiFamilyPath),
in.validateTenancy().ViaField(tenancyPath),
in.validateBlockDeviceMappings().ViaField(blockDeviceMappingsPath),
in.validateTags().ViaField(tagsPath),
)
Expand Down Expand Up @@ -305,3 +307,10 @@ func (in *EC2NodeClassSpec) validateRoleImmutability(originalSpec *EC2NodeClassS
}
return nil
}

func (in *EC2NodeClassSpec) validateTenancy() (errs *apis.FieldError) {
if in.Tenancy != nil {
return in.validateStringEnum(*in.Tenancy, "tenancy", ec2.Tenancy_Values())
}
return nil
}
5 changes: 5 additions & 0 deletions pkg/apis/v1beta1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions pkg/providers/amifamily/resolver.go
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ type LaunchTemplate struct {
AMIID string
InstanceTypes []*cloudprovider.InstanceType `hash:"ignore"`
DetailedMonitoring bool
Tenancy string
EFACount int
CapacityType string
}
Expand Down Expand Up @@ -231,6 +232,7 @@ func (r Resolver) resolveLaunchTemplate(nodeClass *v1beta1.EC2NodeClass, nodeCla
MetadataOptions: nodeClass.Spec.MetadataOptions,
DetailedMonitoring: aws.BoolValue(nodeClass.Spec.DetailedMonitoring),
AMIID: amiID,
Tenancy: aws.StringValue(nodeClass.Spec.Tenancy),
InstanceTypes: instanceTypes,
EFACount: efaCount,
CapacityType: capacityType,
Expand Down
3 changes: 3 additions & 0 deletions pkg/providers/launchtemplate/launchtemplate.go
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,9 @@ func (p *DefaultProvider) createLaunchTemplate(ctx context.Context, options *ami
SecurityGroupIds: lo.Ternary(networkInterfaces != nil, nil, lo.Map(options.SecurityGroups, func(s v1beta1.SecurityGroup, _ int) *string { return aws.String(s.ID) })),
UserData: aws.String(userData),
ImageId: aws.String(options.AMIID),
Placement: &ec2.LaunchTemplatePlacementRequest{
Tenancy: aws.String(options.Tenancy),
},
MetadataOptions: &ec2.LaunchTemplateInstanceMetadataOptionsRequest{
HttpEndpoint: options.MetadataOptions.HTTPEndpoint,
HttpProtocolIpv6: options.MetadataOptions.HTTPProtocolIPv6,
Expand Down
25 changes: 25 additions & 0 deletions pkg/providers/launchtemplate/suite_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2156,6 +2156,31 @@ var _ = Describe("LaunchTemplate Provider", func() {
})
})
})
Context("Tenancy dedicated", func() {
It("should default tenancy was not set", func() {
nodeClass.Spec.AMIFamily = &v1beta1.AMIFamilyAL2
ExpectApplied(ctx, env.Client, nodePool, nodeClass)
pod := coretest.UnschedulablePod()
ExpectProvisioned(ctx, env.Client, cluster, cloudProvider, prov, pod)
ExpectScheduled(ctx, env.Client, pod)
Expect(awsEnv.EC2API.CalledWithCreateLaunchTemplateInput.Len()).To(BeNumerically(">=", 1))
awsEnv.EC2API.CalledWithCreateLaunchTemplateInput.ForEach(func(ltInput *ec2.CreateLaunchTemplateInput) {
Expect(aws.StringValue(ltInput.LaunchTemplateData.Placement.Tenancy)).To(BeNil())
})
})
It("should pass tenancy setting to the dedicated", func() {
nodeClass.Spec.AMIFamily = &v1beta1.AMIFamilyAL2
nodeClass.Spec.Tenancy = aws.String("dedicated")
ExpectApplied(ctx, env.Client, nodePool, nodeClass)
pod := coretest.UnschedulablePod()
ExpectProvisioned(ctx, env.Client, cluster, cloudProvider, prov, pod)
ExpectScheduled(ctx, env.Client, pod)
Expect(awsEnv.EC2API.CalledWithCreateLaunchTemplateInput.Len()).To(BeNumerically(">=", 1))
awsEnv.EC2API.CalledWithCreateLaunchTemplateInput.ForEach(func(ltInput *ec2.CreateLaunchTemplateInput) {
Expect(aws.StringValue(ltInput.LaunchTemplateData.Placement.Tenancy)).To(Equal("dedicated"))
})
})
})
})

// ExpectTags verifies that the expected tags are a subset of the tags found
Expand Down
17 changes: 17 additions & 0 deletions website/content/en/v0.37/concepts/nodeclasses.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,10 @@ spec:
# Optional, configures if the instance should be launched with an associated public IP address.
# If not specified, the default value depends on the subnet's public IP auto-assign setting.
associatePublicIPAddress: true

# Optional, configures tenancy for the instance
# Currently only support default and dedicated
tenancy: default
status:
# Resolved subnets
subnets:
Expand Down Expand Up @@ -1110,6 +1114,19 @@ If a `NodeClaim` requests `vpc.amazonaws.com/efa` resources, `spec.associatePubl
requires that the field is only set to true when configuring an instance with a single ENI at launch. When using this field, it is advised that users segregate their EFA workload to use a separate `NodePool` / `EC2NodeClass` pair.
{{% /alert %}}

## spec.tenancy

A string field that sets the [Dedicated instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/dedicated-instance.html). Currently allowed values are default and dedicated.

```yaml
spec:
tenancy: dedicated
```

By default, EC2 instances run on shared tenancy hardware. This means that multiple AWS accounts can share the same physical hardware.

Dedicated instances are EC2 instances that run on hardware dedicated to a single AWS account. This means that Dedicated Instances are physically isolated at the host hardware level from instances belonging to other AWS accounts, even if that account is associated with a single payer account. However, Dedicated Instances can share hardware with other instances in the same AWS account that are not Dedicated Instances.

## status.subnets
[`status.subnets`]({{< ref "#statussubnets" >}}) contains the resolved `id` and `zone` of the subnets that were selected by the [`spec.subnetSelectorTerms`]({{< ref "#specsubnetselectorterms" >}}) for the node class. The subnets will be sorted by the available IP address count in decreasing order.

Expand Down
Loading