fix(controller): Fix race of csi controller calls on the same volume #588

Lucaber · 2024-09-17T15:39:46Z

Why is this PR required? What issue does it fix?:

When a CreateVolume requests takes longer then 10 seconds the request runs into a client-side timeout and gets retied by kubelet. While processing the new request, the old request is still being canceled on the server-side and the created ZFSVolume CR gets deleted.

1	2
CreateVolume call
ZFSVolume CR created
waiting for volume creation
client timeout + context cancel
	CreateVolume call
	ZFSVolume CR already exists
ZFSVolume CR deleted
Return ERR	Return OK without creating a ZFSVolume

I0916 15:01:49.227721       1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateVolume requests {"accessibility_requirements":{"preferred":[{"segments":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"i04-1-b-node-i041b-0-0","kubernetes.io/os":"linux","m3.services/cluster":"i04-1-b","m3.services/worker-group":"i041b-0","node.systems.mittwald.cloud/placement-group":"i041b-0","openebs.io/nodeid":"i04-1-b-node-i041b-0-0","openebs.io/nodename":"i04-1-b-node-i041b-0-0"}}],"requisite":[{"segments":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"i04-1-b-node-i041b-0-0","kubernetes.io/os":"linux","m3.services/cluster":"i04-1-b","m3.services/worker-group":"i041b-0","node.systems.mittwald.cloud/placement-group":"i041b-0","openebs.io/nodeid":"i04-1-b-node-i041b-0-0","openebs.io/nodename":"i04-1-b-node-i041b-0-0"}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457","parameters":{"compression":"off","csi.storage.k8s.io/pv/name":"pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457","csi.storage.k8s.io/pvc/name":"test-pvc0","csi.storage.k8s.io/pvc/namespace":"default","dedup":"off","fstype":"zfs","poolname":"kluster-pool/i04-1-b-128k","recordsize":"128k","shared":"yes","thinprovision":"yes"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"zfs"}},"access_mode":{"mode":1}}]}
I0916 15:01:58.853180       1 controller.go:289] zfs: trying volume creation kluster-pool/i04-1-b-128k/pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457 on node [i04-1-b-node-i041b-0-0]
I0916 15:01:59.104207       1 volume.go:139] zfs: waiting for volume kluster-pool/i04-1-b-128k/pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457 to be created on nodeid i04-1-b-node-i041b-0-0
I0916 15:02:00.105273       1 volume.go:170] zfs: volume kluster-pool/i04-1-b-128k/pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457 provisioning failed on nodeid i04-1-b-node-i041b-0-0 err: zfs: context deadline reached
I0916 15:02:00.235113       1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateVolume requests {"accessibility_requirements":{"preferred":[{"segments":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"i04-1-b-node-i041b-0-0","kubernetes.io/os":"linux","m3.services/cluster":"i04-1-b","m3.services/worker-group":"i041b-0","node.systems.mittwald.cloud/placement-group":"i041b-0","openebs.io/nodeid":"i04-1-b-node-i041b-0-0","openebs.io/nodename":"i04-1-b-node-i041b-0-0"}}],"requisite":[{"segments":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"i04-1-b-node-i041b-0-0","kubernetes.io/os":"linux","m3.services/cluster":"i04-1-b","m3.services/worker-group":"i041b-0","node.systems.mittwald.cloud/placement-group":"i041b-0","openebs.io/nodeid":"i04-1-b-node-i041b-0-0","openebs.io/nodename":"i04-1-b-node-i041b-0-0"}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457","parameters":{"compression":"off","csi.storage.k8s.io/pv/name":"pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457","csi.storage.k8s.io/pvc/name":"test-pvc0","csi.storage.k8s.io/pvc/namespace":"default","dedup":"off","fstype":"zfs","poolname":"kluster-pool/i04-1-b-128k","recordsize":"128k","shared":"yes","thinprovision":"yes"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"zfs"}},"access_mode":{"mode":1}}]}
I0916 15:02:00.449153       1 volume.go:221] zfs: deleted the volume pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457
I0916 15:02:00.449299       1 controller.go:466] created the volume kluster-pool/i04-1-b-128k/pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457 on node i04-1-b-node-i041b-0-0
I0916 15:02:00.553778       1 grpc.go:81] GRPC response: {"volume":{"accessible_topology":[{"segments":{"openebs.io/nodeid":"i04-1-b-node-i041b-0-0"}}],"capacity_bytes":5368709120,"volume_context":{"openebs.io/cas-type":"localpv-zfs","openebs.io/poolname":"kluster-pool/i04-1-b-128k"},"volume_id":"pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457"}}

What this PR does?: Adds a per volume mutex to csi calls to prevent simultaneous requests

Does this PR require any upgrade changes?: No

If the changes in this PR are manually verified, list down the scenarios covered::

Any additional information for your reviewer? :
Mention if this PR is part of any design or a continuation of previous PRs

Checklist:

Fixes #
PR Title follows the convention of <type>(<scope>): <subject>
Has the change log section been updated?
Commit has unit tests
Commit has integration tests
(Optional) Are upgrade changes included in this PR? If not, mention the issue/PR to track:
(Optional) If documentation changes are required, which issue on https://github.com/openebs/website is used to track them:

codecov-commenter · 2024-09-21T18:01:29Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.99%. Comparing base (8c402d3) to head (27ec79f).
Report is 6 commits behind head on develop.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop     #588   +/-   ##
========================================
  Coverage    95.99%   95.99%           
========================================
  Files            1        1           
  Lines          574      574           
========================================
  Hits           551      551           
  Misses          19       19           
  Partials         4        4

Flag	Coverage Δ
bddtests	`95.99% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

niladrih · 2024-09-25T10:40:37Z

Hmm... this is upsetting. I suppose all of the tests on our Go repos need to use the -race flag on CI pipeline.
I've created this ticket: openebs/openebs#3784

tiagolobocastro

Thanks!
I wonder if we're missing this in other places...

pkg/driver/controller.go

Lucaber · 2024-09-25T11:46:51Z

Hmm... this is upsetting. I suppose all of the tests on our Go repos need to use the -race flag on CI pipeline. I've created this ticket: openebs/openebs#3784

This isn't a go data race, -race can't detect this issue.
To reproduce this issue I created 50 PVCs with WaitForFirstConsumer and then started a single pod with all 50 volumes to create a high load on both kubelet and openebs.

tiagolobocastro

Thanks, lgtm, though I still somehow wonder if returning an error would be a safer approach.

Abhinandan-Purkait · 2024-10-21T06:57:22Z

@Lucaber Please rebase your branch. @tiagolobocastro Are we good to take this in now?

tiagolobocastro · 2024-10-21T09:47:50Z

@Lucaber Please rebase your branch. @tiagolobocastro Are we good to take this in now?

Yeah I had approved already

Abhinandan-Purkait · 2024-10-23T04:03:33Z

@Lucaber Mark the conversations as resolved if you don't think there is anything more to do before we can merge this.

When a CreateVolume requests takes longer then 10 seconds the request runs into a client-side timeout and gets retied by kubelet. While processing the new request, the old request is still being canceled on the server-side and the created ZFSVolume CR gets deleted. 1|2 CreateVolume| ZFSVolume CR created| waiting for volume creation| client timeout + context cancel| |CreateVolume |ZFSVolume CR already exists ZFSVolume CR deleted| Return ERR|Return OK without creating a ZFSVolume ``` I0916 15:01:49.227721 1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateVolume requests {"accessibility_requirements":{"preferred":[{"segments":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"i04-1-b-node-i041b-0-0","kubernetes.io/os":"linux","m3.services/cluster":"i04-1-b","m3.services/worker-group":"i041b-0","node.systems.mittwald.cloud/placement-group":"i041b-0","openebs.io/nodeid":"i04-1-b-node-i041b-0-0","openebs.io/nodename":"i04-1-b-node-i041b-0-0"}}],"requisite":[{"segments":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"i04-1-b-node-i041b-0-0","kubernetes.io/os":"linux","m3.services/cluster":"i04-1-b","m3.services/worker-group":"i041b-0","node.systems.mittwald.cloud/placement-group":"i041b-0","openebs.io/nodeid":"i04-1-b-node-i041b-0-0","openebs.io/nodename":"i04-1-b-node-i041b-0-0"}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457","parameters":{"compression":"off","csi.storage.k8s.io/pv/name":"pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457","csi.storage.k8s.io/pvc/name":"test-pvc0","csi.storage.k8s.io/pvc/namespace":"default","dedup":"off","fstype":"zfs","poolname":"kluster-pool/i04-1-b-128k","recordsize":"128k","shared":"yes","thinprovision":"yes"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"zfs"}},"access_mode":{"mode":1}}]} I0916 15:01:58.853180 1 controller.go:289] zfs: trying volume creation kluster-pool/i04-1-b-128k/pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457 on node [i04-1-b-node-i041b-0-0] I0916 15:01:59.104207 1 volume.go:139] zfs: waiting for volume kluster-pool/i04-1-b-128k/pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457 to be created on nodeid i04-1-b-node-i041b-0-0 I0916 15:02:00.105273 1 volume.go:170] zfs: volume kluster-pool/i04-1-b-128k/pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457 provisioning failed on nodeid i04-1-b-node-i041b-0-0 err: zfs: context deadline reached I0916 15:02:00.235113 1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateVolume requests {"accessibility_requirements":{"preferred":[{"segments":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"i04-1-b-node-i041b-0-0","kubernetes.io/os":"linux","m3.services/cluster":"i04-1-b","m3.services/worker-group":"i041b-0","node.systems.mittwald.cloud/placement-group":"i041b-0","openebs.io/nodeid":"i04-1-b-node-i041b-0-0","openebs.io/nodename":"i04-1-b-node-i041b-0-0"}}],"requisite":[{"segments":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"i04-1-b-node-i041b-0-0","kubernetes.io/os":"linux","m3.services/cluster":"i04-1-b","m3.services/worker-group":"i041b-0","node.systems.mittwald.cloud/placement-group":"i041b-0","openebs.io/nodeid":"i04-1-b-node-i041b-0-0","openebs.io/nodename":"i04-1-b-node-i041b-0-0"}}]},"capacity_range":{"required_bytes":5368709120},"name":"pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457","parameters":{"compression":"off","csi.storage.k8s.io/pv/name":"pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457","csi.storage.k8s.io/pvc/name":"test-pvc0","csi.storage.k8s.io/pvc/namespace":"default","dedup":"off","fstype":"zfs","poolname":"kluster-pool/i04-1-b-128k","recordsize":"128k","shared":"yes","thinprovision":"yes"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"zfs"}},"access_mode":{"mode":1}}]} I0916 15:02:00.449153 1 volume.go:221] zfs: deleted the volume pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457 I0916 15:02:00.449299 1 controller.go:466] created the volume kluster-pool/i04-1-b-128k/pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457 on node i04-1-b-node-i041b-0-0 I0916 15:02:00.553778 1 grpc.go:81] GRPC response: {"volume":{"accessible_topology":[{"segments":{"openebs.io/nodeid":"i04-1-b-node-i041b-0-0"}}],"capacity_bytes":5368709120,"volume_context":{"openebs.io/cas-type":"localpv-zfs","openebs.io/poolname":"kluster-pool/i04-1-b-128k"},"volume_id":"pvc-eb11c446-f26c-4333-b3da-4ecb9c5b2457"}} ``` Signed-off-by: Luca Berneking <l.berneking@mittwald.de>

Signed-off-by: Luca Berneking <l.berneking@mittwald.de>

Abhinandan-Purkait · 2024-11-04T09:19:34Z

Merging this once the CI passes. Thanks for the contribution @Lucaber

Abhinandan-Purkait requested review from tiagolobocastro, avishnu, sinhaashish, Abhinandan-Purkait and niladrih September 21, 2024 16:51

Abhinandan-Purkait approved these changes Sep 25, 2024

View reviewed changes

tiagolobocastro approved these changes Sep 25, 2024

View reviewed changes

pkg/driver/controller.go Outdated Show resolved Hide resolved

tiagolobocastro reviewed Sep 25, 2024

View reviewed changes

pkg/driver/controller.go Outdated Show resolved Hide resolved

Lucaber force-pushed the fix/controller-csi-race branch 2 times, most recently from d2e4509 to 32b9de3 Compare September 25, 2024 12:42

Lucaber requested review from tiagolobocastro and Abhinandan-Purkait October 4, 2024 14:09

tiagolobocastro approved these changes Oct 4, 2024

View reviewed changes

Lucaber force-pushed the fix/controller-csi-race branch from b3e82b5 to 751deb2 Compare October 4, 2024 14:37

Abhinandan-Purkait approved these changes Oct 21, 2024

View reviewed changes

Lucaber force-pushed the fix/controller-csi-race branch from 751deb2 to 6690f5c Compare October 21, 2024 10:06

Lucaber added 3 commits November 4, 2024 14:45

Add LockVolumeWithSnapshot function to prevent future deadlocks

bee0aa6

Signed-off-by: Luca Berneking <l.berneking@mittwald.de>

Use single mutex for volume locks to prevent memory leak

27ec79f

Signed-off-by: Luca Berneking <l.berneking@mittwald.de>

Abhinandan-Purkait force-pushed the fix/controller-csi-race branch from 6690f5c to 27ec79f Compare November 4, 2024 09:15

Abhinandan-Purkait merged commit b2f57b9 into openebs:develop Nov 4, 2024
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(controller): Fix race of csi controller calls on the same volume #588

fix(controller): Fix race of csi controller calls on the same volume #588

Lucaber commented Sep 17, 2024 •

edited

Loading

codecov-commenter commented Sep 21, 2024 •

edited

Loading

niladrih commented Sep 25, 2024 •

edited

Loading

tiagolobocastro left a comment

Lucaber commented Sep 25, 2024

tiagolobocastro left a comment

Abhinandan-Purkait commented Oct 21, 2024

tiagolobocastro commented Oct 21, 2024

Abhinandan-Purkait commented Oct 23, 2024

Abhinandan-Purkait commented Nov 4, 2024

fix(controller): Fix race of csi controller calls on the same volume #588

fix(controller): Fix race of csi controller calls on the same volume #588

Conversation

Lucaber commented Sep 17, 2024 • edited Loading

codecov-commenter commented Sep 21, 2024 • edited Loading

Codecov Report

niladrih commented Sep 25, 2024 • edited Loading

tiagolobocastro left a comment

Choose a reason for hiding this comment

Lucaber commented Sep 25, 2024

tiagolobocastro left a comment

Choose a reason for hiding this comment

Abhinandan-Purkait commented Oct 21, 2024

tiagolobocastro commented Oct 21, 2024

Abhinandan-Purkait commented Oct 23, 2024

Abhinandan-Purkait commented Nov 4, 2024

Lucaber commented Sep 17, 2024 •

edited

Loading

codecov-commenter commented Sep 21, 2024 •

edited

Loading

niladrih commented Sep 25, 2024 •

edited

Loading