Ceph CSI with Nomad 1.1.0-rc1

[chewrocca]

Nomad version

Nomad v1.1.0-rc1 (c16a285)

Operating system and Environment details

Ubuntu20.04 LTS

Issue

When creating a volume with nomad volume create volume.hcl it creates the image on the desired pool in Ceph. However, when I try to run a job using that volume it fails saying failed to setup alloc: pre-run hook "csi_hook" failed: unknown volume attachment mode:

The output of nomad volume status ceph-mysql seems to indicate there is no access mode or attachment mode even though that is set in volume.hcl

nomad volume status ceph-mysql
ID                   = ceph-mysql
Name                 = ceph-mysql
External ID          = 0001-0024-357d556c-b7dc-11eb-8bda-cfc8b3ffbe43-0000000000000005-d0ca3eb8-b7fa-11eb-962a-0242ac110002
Plugin ID            = ceph-csi
Provider             = rbd.csi.ceph.com
Version              = v3.3.1
Schedulable          = true
Controllers Healthy  = 1
Controllers Expected = 1
Nodes Healthy        = 1
Nodes Expected       = 1
Access Mode          = <none>
Attachment Mode      = <none>
Mount Options        = <none>
Namespace            = default

Allocations
No allocations placed

Reproduction steps

1. spin up Ceph
2. create pool, permissions, etc
3. run Nomad plugin jobs for controller and nodes
4. create volume on Ceph using nomad volume create volume.hcl
5. attempt to run MySQL Nomad job

Expected Result

I'd expect the output of nomad volume status ceph-mysql to show access and attachment modes

Actual Result

nomad volume status ceph-mysql
ID                   = ceph-mysql
Name                 = ceph-mysql
External ID          = 0001-0024-357d556c-b7dc-11eb-8bda-cfc8b3ffbe43-0000000000000005-d0ca3eb8-b7fa-11eb-962a-0242ac110002
Plugin ID            = ceph-csi
Provider             = rbd.csi.ceph.com
Version              = v3.3.1
Schedulable          = true
Controllers Healthy  = 1
Controllers Expected = 1
Nodes Healthy        = 1
Nodes Expected       = 1
Access Mode          = <none>
Attachment Mode      = <none>
Mount Options        = <none>
Namespace            = default

Allocations
No allocations placed

Job file (if appropriate)

cat volume.hcl
id        = "ceph-mysql"
name      = "ceph-mysql"
type      = "csi"
plugin_id = "ceph-csi"

capacity_min = "5GB"
capacity_max = "10GB"

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

#mount_options {
#  fs_type     = "ext4"
#  mount_flags = ["rw"]
#}

secrets {
  userID  = "myPoolAdmin"
  userKey = "<my-redacted-userKey>"
}

parameters {
  clusterID     = "<my-cluster-id>"
  pool          = "myPool"
  imageFeatures = "layering"
}

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[sundbry]

Edit: I had the old nomad client (1.0.4)

Make sure you set access_mode and attachment_mode in the job specification in addition to the capabilities in the volume registration.

[chewrocca]

@sundbry This makes sense. Thank you.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.