NodeUnpublishVolume error after a pod is force deleted

[sba30]

/kind bug

What happened?
After a pod has been force deleted, We see system load very high on the eks worker. And in the efs-csi-node logs we see this error below referencing the pod which does not exist anymore.

NodeUnpublishVolume: called with args volume_id:"fs-xxxxxx::fsap-xxxxx" target_path:"/var/lib/kubelet/pods/xyxxyz/volumes/kubernetes.io~csi/abc-efs/mount"

What you expected to happen?
Load not increase and this NodeUnpublishVolume not in the logs

How to reproduce it (as minimally and precisely as possible)?
Force delete a pod which uses a pvc, and review the esf-csi-nodes

Anything else we need to know?:

Environment

• Kubernetes version (use kubectl version):
  v1.26.7-eks-8ccc7ba

• Driver version:
  aws-efs-csi-driver:v1.5.6
  Please also attach debug logs to help us better diagnose

• Instructions to gather debug logs can be found here

[seanzatzdev-amazon]

Hello @sba30 could you please provide the full debug logs following this as well as let us know how you have installed the driver?

[sba30]

Hi, so on review off this issue #112, we were able to get past the issue, by exec'ing onto the pod reporting the issue, the UID of the pod seems to be an old pod and not the active one, so when we ran

umount -l <target path>

then the NodeUnpublishVolume does not appear in the efs logs anymore and the load comes down.

unsure if theres a permanent fix here or not, but this workaround has worked for us.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.