docs: add RFC on making expiration forceful

[njtran]

Fixes #N/A

Description
This proposes making Expiration forceful, allowing cluster admins to have a guaranteed way to understand the maximum node lifetime of the node.

How was this change tested?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[njtran]

Hey all, I figure this is a long shot, but as you all were up-votes on aws/karpenter-provider-aws#1884, I wanted to see if any of you had thoughts on this RFC, and know the use-cases for Graceful Expiration.

@andrebikeev @agrrh @karma-git @funes79 @FernandoMiguel @duboisf @diranged @creosonic @gbvanrenswoude @universam1 @philslab-ninja @round-corgi @mmukarram @grandich

[FernandoMiguel]

Thanks for the ping.
I'll do my best to take a look tomorrow.

[GnatorX]

Would calling Graceful "Launch before Terminate" and Forceful as "Terminate and launch" make more sense? Both flow is still graceful they just order the node creation differently. AWS already use these terminology https://docs.aws.amazon.com/autoscaling/ec2/userguide/instance-maintenance-policy-overview-and-considerations.html

[njtran]

I think there's value in having one-word terms to map to the phrase, but I do agree those are pretty clear. There's another mechanism in between called Eventual that's not called out here that is harder to explain in those terms as well.

[FernandoMiguel]

there seems to be a design issue here.
when using controllers like ebs csi plugin, which work as daemonset, if you drain it before removing the EBS volumes, the volumes wont be detached.

[GnatorX]

Wouldn't deletion of pods that asked for the volumes kick off the CSI del to remove those volumes?

[jmdeal]

If the ebs-csi-node pod tolerates Karpenter's disruption taint and is drained early, this could result in volumes still being attached. This shouldn't happen in this specific case since it has a preStop hook that waits for all volumes to be detached, though this can still be problematic if its terminationGracePeriodSeconds is exceeded before all volumes are detached. This should also apply once the instance is terminating if graceful node termination has been configured (though there's currently an issue since Karpenter removes the Node object before the instance is terminated).

[FernandoMiguel]

this is a current issue. blocking many Orgs.
kubernetes-sigs/aws-ebs-csi-driver#1955 (comment)
So whatever we design here needs to take in consideration who currently Karpenter works and plan for the future, or I fear we may make the issue worst by assuming certain things

[FernandoMiguel]

I like option 1

[GnatorX]

Overall lgtm. I only have questions for myself

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: GnatorX, njtran

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [njtran]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[njtran]

For those I've pinged, hoping to close out the RFC this week. Please let me know if you have questions.

[jmdeal]

/lgtm

[coveralls]

Pull Request Test Coverage Report for Build 9589288784

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 480 unchanged lines in 22 files lost coverage.
• Overall coverage increased (+2.2%) to 80.526%

---

Files with Coverage Reduction	New Missed Lines	%
pkg/controllers/disruption/consolidation.go	2	90.2%
pkg/controllers/disruption/emptynodeconsolidation.go	2	95.0%
pkg/controllers/disruption/expiration.go	2	93.1%
pkg/apis/v1beta1/nodepool.go	4	95.24%
pkg/utils/pretty/pretty.go	5	86.84%
pkg/cloudprovider/fake/cloudprovider.go	6	96.63%
pkg/utils/node/node.go	7	76.92%
pkg/controllers/disruption/singlenodeconsolidation.go	7	81.82%
pkg/cloudprovider/types.go	11	85.8%
pkg/controllers/disruption/helpers.go	12	89.94%

Totals
Change from base Build 9392303085:	2.2%
Covered Lines:	8324
Relevant Lines:	10337

---

💛 - Coveralls