Optimize LCH Behavior During Termination Events

[torredil]

Issue Description

Currently, the CSI node pod utilizes OS graceful shutdown time during termination events by waiting for volumes on a node to be detached because Kubelet's node shutdown manager does not consider volume cleanup as a signal and tends to prematurely terminate the driver pod, which can result in delayed attachments.

Ideally, DetachVolume calls should complete before TerminateInstances is invoked - however, in scenarios where TerminateInstances is called by autoscalers such as Karpenter, it is undesirable for the lifecycle hook to wait for detachments to succeed in the backend because DetachVolume requests made after TerminateInstances will wait for the TerminateInstances workflow to complete.

Additional Context

According to the external attacher docs, checking for the presence of DeletionTimestamp in VA objects can be used to ensure that volumes have been successfully unmounted from k8s perspective (which is more reliable than scanning for mounts on the host).

Also, kubelet sets a NotReady condition on the node with the reason set to "node is shutting down", which can be used to determine node status to stop waiting on detaches in specifically the scenario described above.

[levanlongktmt]

@torredil I guess this is reason for my problem at #1665 (comment)

[primeroz]

Is this also the reason for #1955 ?

[torredil]

@primeroz No, the root cause for #1955 was identified as a regression introduced by a breaking change in Karpenter related to taints: kubernetes-sigs/karpenter#508 - that issue was fixed by #1969 and released in driver v1.29.0 (the corresponding add-on release is currently in progress and is expected to be available by March 29)

This specific issue was created to address a scenario / race condition where executing the prestop lifecycle hook is undesirable based on EC2 API behavior - today, DetachVolume requests wait for the TerminateInstances workflow to complete before proceeding. That is to say, if Karpenter has already called TerminateInstances there is no benefit in executing the lifecycle hook.

The optimal solution here would be to make Karpenter aware of volume teardown before invoking TerminateInstances as part of the node drain process. This approach would eliminate the need for executing a lifecycle hook in the first place and prevent delayed attachments as a result of termination events.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale