CSI plugin expected controllers exceeds actual controller count

[iSchluff]

Nomad version

Nomad v1.3.0-beta.1 (2eba643)

Operating system and Environment details

Ubuntu 20.04.4 on x64

Issue

Due to operator error I registered csi nodes as controllers, therefore my expected controller count on the plugin is now too high.

#  nomad plugin status
Container Storage Interface
ID        Provider             Controllers Healthy/Expected  Nodes Healthy/Expected
ceph-csi  cephfs.csi.ceph.com  1/4                           3/3

Nomad still shows the plugin as healthy and the volumes as schedulable, however waiting for plugin healthy via terraform fails:

data "nomad_plugin" "ceph" {
  plugin_id        = "ceph-csi"
  wait_for_healthy = true
}

2022-04-25T12:25:32.941+0200 [TRACE] provider.terraform-provider-nomad_v1.4.16_x4: plugin received interrupt signal, ignoring: count=2 timestamp=2022-04-25T12:25:32.941+0200
2022-04-25T12:25:33.009+0200 [DEBUG] provider.terraform-provider-nomad_v1.4.16_x4: 2022/04/25 12:25:33 [DEBUG] Getting plugin "ceph-csi"...
2022-04-25T12:25:33.065+0200 [DEBUG] provider.terraform-provider-nomad_v1.4.16_x4: 2022/04/25 12:25:33 [DEBUG] plugin ceph-csi not yet healthy: 1/4 controllers healthy  3/3 nodes healthy
2022-04-25T12:25:33.065+0200 [DEBUG] provider.terraform-provider-nomad_v1.4.16_x4: 2022/04/25 12:25:33 [TRACE] Waiting 10s before next try
^C2022-04-25T12:25:34.397+0200 [TRACE] provider.terraform-provider-nomad_v1.4.16_x4: plugin received interrupt signal, ignoring: count=3 timestamp=2022-04-25T12:25:34.397+0200
2022-04-25T12:25:35.967+0200 [TRACE] dag/walk: vertex "module.nomad-registry.nomad_volume.registry (expand)" is waiting for "module.nomad-registry.data.nomad_plugin.ceph (expand)"
2022-04-25T12:25:36.275+0200 [TRACE] dag/walk: vertex "module.nomad-registry (close)" is waiting for "module.nomad-registry.nomad_volume.registry (expand)"
2022-04-25T12:25:36.648+0200 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/nomad\"] (close)" is waiting for "module.nomad-registry.nomad_volume.registry (expand)"
2022-04-25T12:25:37.349+0200 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/nomad\"] (close)"

Is it possible to reduce the expected controller count without recreating the cluster?

Reproduction steps

Start more csi controllers than you need, stop some of them.

Expected Result

expected controller count should probably never exceed 1?

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[tgross]

Hi @iSchluff! I want to make sure I can reproduce this scenario accurately:

Due to operator error I registered csi nodes as controllers, therefore my expected controller count on the plugin is now too high.
...
Start more csi controllers than you need, stop some of them.

So you had a job that was registered as a controller, and then changed it to node and re-ran? Did the old controller allocations get replaced entirely by node allocations or did they get in-place updates?

[iSchluff]

Yes exactly, I accidentally ran node tasks as controller tasks. The old allocations got replaced by an in-place job update.
Just for completeness: I was already running the tasks as nodes before and replaced the csi configuration and then went back.
So basically in-place node>controller>node

[tgross]

Ok, thank you. So there's likely two bugs here: the counts aren't being reset properly, but also we should have replaced the tasks entirely in that case and not done an in-place update. I'll see if I can put together a quick repro and report back.

[tgross]

I was able to reproduce the non-destructive update pretty easily just by switching the csi_plugin.type. #12774 should fix this and prevent the invariants the counts expect from being violated.

[tgross]

I've merged that fix and it'll ship in the GA release of Nomad 1.3.0. Thanks for opening this issue @iSchluff!

[iSchluff]

thanks for the quick response

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.