Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pod in Terminating State won't Unmount #118

Open
washcycle opened this issue May 9, 2023 · 1 comment
Open

Pod in Terminating State won't Unmount #118

washcycle opened this issue May 9, 2023 · 1 comment

Comments

@washcycle
Copy link
Contributor

Mounted storage can get into a state where the weed mount isn't unmounted when the pod is terminating.

I don't have a reproducibe example right now, but wanted to document the issue still.

Work around is to go into the k8s node and kill the pvc mount there.

root@aspen7:/# ps -aux | grep pvc-cebb2309-cdaa-4edd-a81d-bc75e6b5b15d
root     1334193  0.0  0.0 774944 54492 ?        Sl   May08   0:21 weed -logtostderr=true mount -dirAutoCreate=true -umask=000 -dir=/var/lib/k0s/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/a28177b3447eebd81ee4b8be4305
7212220e657c2f1e8f7cbc696edbc3d8fb4f/globalmount -localSocket=/tmp/seaweedfs-mount-1226006495.sock -cacheDir=/var/cache/seaweedfs/pvc-cebb2309-cdaa-4edd-a81d-bc75e6b5b15d -collectionQuotaMB=25600 -concurrentWriters=15 -collectio
n=pvc-cebb2309-cdaa-4edd-a81d-bc75e6b5b15d -filer=seaweedfs-filer-client.seaweedfs:8888 -cacheCapacityMB=1024 -filer.path=/buckets/pvc-cebb2309-cdaa-4edd-a81d-bc75e6b5b15d

sudo kill 1334193

The pod will then terminate.

I wonder if there is something we can add to the csi driver that checks for terminating pods and will kill the seaweedfs mount, i'm not super good with the Go k8s stack, but perhaps inspiration from some other projects that have solved this would help.

@chrislusf
Copy link
Contributor

Check whether NodeUnpublishVolume or NodeUnstageVolume is invoked in
https://github.com/seaweedfs/seaweedfs-csi-driver/blob/master/pkg/driver/nodeserver.go

Maybe add some println.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants