After removing aws-ebs0 from the cluster deleting nodes fails

[analytically]

Nomad version

Nomad v0.11.2 (807cfeb)

Operating system and Environment details

Amazon Linux 2, c5.4xlarge

Issue

After removing the aws-ebs0 plugin from our cluster, we're seeing the client list with ineligble nodes grow unbounded without removing nodes.

The logs have:
nomad.fsm: DeleteNode failed: error="csi plugin delete failed: csi_plugins missing pl ugin aws-ebs0"

[analytically]

Similar to #8100

[tgross]

Hi @analytically can you clarify for me the symptom you're seeing here:

the client list with ineligble nodes grow unbounded without removing nodes.

Nomad nodes are being marked as ineligible for placing workloads? Or do you mean you can't schedule CSI workloads on those nodes?

Similar to #8100

That issue is about volume claim GC, which should be unrelated.

[analytically]

No this is also the volume claim GC (Nomad Nodes GC)

[tgross]

There's a difference between "volume claim GC" (which is the cleanup of claims that an allocation has on a volume) and a "node GC" (which is the cleanup of Nomad clients).

I'm still not clear what the symptom you're seeing it: Nomad nodes are being marked as ineligible for placing workloads? Or do you mean you can't schedule CSI workloads on those nodes?

[tgross]

Some follow-up data which isn't quite the same thing but could be related. If we try to delete a job that hasn't registered its plugin, that results in the following error state:

2020-07-29T14:54:24.417Z [ERROR] nomad.fsm: DeleteJob failed: error="deleting job from plugin: plugin missing: aws-ebs0 <nil>"
2020-07-29T14:54:24.417Z [ERROR] nomad.fsm: deregistering job failed: error="deleting job from plugin: plugin missing: aws-ebs0 <nil>"

[tgross]

Should be fixed with #8619

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.