fix: Ensure persistent volumes are detached before deleting node

[AndrewSirenko]

Fixes #N/A

Description

Fixes 6+ minute delays for disrupted EBS-Backed stateful workloads when starting on their new node.

For more context see RFC for solving 6+ minute delays for disrupted stateful workloads

TLDR:

In order for a stateful pod to smoothly migrate from terminating node to new node...

0. Consolidation event starts
1. Stateful pods must terminate
2. EBS CSI Node pod must unmount all filesystems (NodeUnpublish & NodeUnstage RPCs)
3. EBS CSI Controller pod must detach all volumes from instance
4. Karpenter terminates EC2 Instance
5. Karpenter ensures Node object deleted from Kubernetes

Problems:
A. If 2 doesn't happen, today there's a 6+ minute delay in stateful pod migration because Kubernetes is afraid volume still attached and mounted to instance (6+ min delay)
B. If 3 doesn't happen, the new stateful pod can't start until consolidated instance is terminated which auto-detaches volumes (1+ min delay)

Solution:
Wait for volumeattachment objects associated with drainable pods & non-multi-attach volumes before deleting the node.

How was this change tested?

Manual: Create statefulset + nodepool. Have nodes expire every 3 minutes. Check that stateful pods migrate to new node and start running in under a minute.

Also tested that we do not block deletion when there are stateful workloads that tolerate all taints, or Node terminationGracePeriod elapsed.

Additional Notes

Note 1: Must add read permissions for volumeattachments to clusterrole-core.yaml in karpenter-provider-aws

  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses", "csinodes", "volumeattachments"]
    verbs: ["get", "watch", "list"]

Note 2: Separate PR to add e2e tests in karpenter-provider-aws: aws/karpenter-provider-aws#6484

Note 3: It was decided in the RFC we will block node deletion for all volumeattachments, regardless of CSI Driver. In the future, we may decide to inject a list of CSI Drivers via the cloud provider instead.

Note 4: There might be some rare cases where EBS CSI Node pod can get killed before it unmounts volumes. The solution would be karpenter (or some reliable automation) tainting node with nodeshutdown:NoExecute once node is terminated, as discussed in RFC. In design meeting consensus was that this could be added later if customers run into it.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[AndrewSirenko]

In the future we should consider adding an e2e test to karpenter-provider-aws that tests that a stateful workload on a consolidating node is migrated and starts running on second node < 6 minutes to prevent regressions. @jmdeal mentioned that EBS CSI Driver is already installed on e2e environment. Thoughts?

[coveralls]

Pull Request Test Coverage Report for Build 9370530435

Details

• 49 of 67 (73.13%) changed or added relevant lines in 5 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.04%) to 77.922%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/controllers/node/termination/terminator/terminator.go	18	20	90.0%
pkg/operator/operator.go	0	3	0.0%
pkg/utils/node/node.go	7	12	58.33%
pkg/controllers/node/termination/controller.go	15	23	65.22%

Totals
Change from base Build 9356953903:	-0.04%
Covered Lines:	8319
Relevant Lines:	10676

---

💛 - Coveralls

[github-actions]

This PR has been inactive for 14 days. StaleBot will close this stale PR after 14 more days of inactivity.

[jmdeal]

/remove-lifecycle stale

[jonathan-innis]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AndrewSirenko, jonathan-innis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jonathan-innis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jonathan-innis]

/hold Wait for E2E tests to complete in the CloudProvider repo

[jonathan-innis]

/lgtm

[jonathan-innis]

/unhold e2es are passing in the AWS provider repo

[AndrewSirenko]

/unhold

^^