CSI: single access_mode is not enforced within a job

[tgross]

A volume that has access_mode = "single-node-reader-only" or access_mode = "single-node-writer" is feasibility checked during scheduling to ensure that only a single reader or writer claim exists. However, because feasibility checking is done one alloc at a time before the plan is written, a job that's misconfigured to have count > 1 that mounts one of these volumes will pass feasibility checking. The first allocation will be placed, but if the storage provider enforces its invariants the second and later allocations will fail after placement on the client.

To reproduce, use this volume specification with the hostpath plugin (with the UUID value set from the ID returned from csc):

id = "test-volume0"
name = "test-volume0"
type = "csi"
external_id = "$UUID""
plugin_id = "hostpath-plugin0"
access_mode = "single-node-reader-only"
attachment_mode = "file-system"

And this job specification:

job "example" {
  datacenters = ["dc1"]

  group "cache" {
    count = 2

    volume "volume0" {
      type      = "csi"
      source    = "test-volume0"
      read_only = true
    }

    task "redis" {
      driver = "docker"
      config {
        image = "redis:3.2"
      }

      volume_mount {
        volume      = "volume0"
        destination = "${NOMAD_ALLOC_DIR}/volume0"
      }
    }
  }
}

Note that the hostpath plugin is a toy plugin and does not enforce its invariants, so you'll see two allocations merrily scheduled to use the volume that's been marked for single reader.

Discovered this while writing tests for #10136

[tgross]

Noting that this will be fixed by the solution proposed here: #8609 (comment)

Edit: nope! this issue is a schedule-time problem and needs to be fixed separately from #8609

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.