Cinder-CSI Plugin reports 0 healthy nodes/controllers after node decommissioning

[ksitnik-tc]

Nomad version

Nomad v1.0.2

Issue

We run our client nodes with the cinder-csi plugin for persistent storage.

When we decommission client nodes, Nomad reports plugin health status of 0.

For example, prior to stopping the clients,

nomad plugin status
Container Storage Interface
ID        Provider                  Controllers Healthy/Expected  Nodes Healthy/Expected
dev  cinder.csi.openstack.org  6/6                           6/6

After removing the old clients,

nomad plugin status
Container Storage Interface
ID        Provider                  Controllers Healthy/Expected  Nodes Healthy/Expected
dev  cinder.csi.openstack.org  4/0                           4/0

Re-applying the cinder-csi plugin job does not remedy this issue.

[apollo13]

Probably related/similar cause to #9810

[tgross]

In #11758 (comment) I've noted what's probably the cause of and fix for this issue:

The plugin counts don't accurately match the state of allocations. I'm working up a patch the resolves plugin counts in a similar fashion to what we did for volume claims in #11890

[tgross]

I've just opened #12027 which should close out this issue.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.