csi: support Secrets parameter in CSI RPCs #7923

tgross · 2020-05-11T17:55:41Z

CSI plugins can require credentials for some publishing and unpublishing workflow RPCs. Secrets are configured at the time of volume registration, stored in the volume struct, and then passed around as an opaque map by Nomad to the plugins.

I've left some comments in this PR as I was working through it to help us implement #7771 and #7670 later, but we're running out of time in the 0.11.2 window to complete all three and make sure it's tested completely.

tgross · 2020-05-11T19:28:47Z

I don't have a Ceph setup to validate this end-to-end yet, but fortunately some of the other plugins log that they got the secrets (sometimes just insecurely printing them!) if you send them, even if the plugin doesn't use it.

Hostpath plugin node publish operations:

I0511 17:58:59.728882       1 server.go:117] GRPC call: /csi.v1.Node/NodeStageVolume
I0511 17:58:59.728899       1 server.go:118] GRPC request: {"secrets":"***stripped***","staging_target_path":"/csi/staging/test-volume0/rw-file-system-single-node-writer","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_id":"17264bf3-93b1-11ea-8b80-0242ac110002"}
I0511 17:58:59.730818       1 server.go:123] GRPC response: {}
I0511 17:58:59.734539       1 server.go:117] GRPC call: /csi.v1.Node/NodePublishVolume
I0511 17:58:59.734551       1 server.go:118] GRPC request: {"secrets":"***stripped***","staging_target_path":"/csi/staging/test-volume0/rw-file-system-single-node-writer","target_path":"/csi/per-alloc/8c97cc5a-3e03-2016-b0fc-bdc5f4001c6b/test-volume0/rw-file-system-single-node-writer","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_id":"17264bf3-93b1-11ea-8b80-0242ac110002"}

EBS register a volume:

I0511 18:08:00.118455       1 controller.go:324] ValidateVolumeCapabilities: called with args {VolumeId:vol-024d9fe7b511b9be6 VolumeContext:map[] VolumeCapabilities:[mount:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[] Secrets:map[secretsauce:xyzzy] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}

EBS controller publish volume:

I0511 18:09:21.052749       1 controller.go:229] ControllerPublishVolume: called with args {VolumeId:vol-024d9fe7b511b9be6 NodeId:i-087bb3ed4f5954a3a VolumeCapability:mount:<> access_mode:<mode:SINGLE_NODE_WRITER >  Readonly:false Secrets:map[secretsauce:xyzzy] VolumeContext:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}

EBS node publish (note that node unpublish has no secrets options):

I0511 18:09:24.699656       1 node.go:94] NodeStageVolume: called with args {VolumeId:vol-024d9fe7b511b9be6 PublishContext:map[devicePath:/dev/xvdba] StagingTargetPath:/csi/staging/ebs-vol0/rw-file-system-single-node-writer VolumeCapability:mount:<> access_mode:<mode:SINGLE_NODE_WRITER >  Secrets:map[secretsauce:xyzzy] VolumeContext:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0511 18:09:24.699794       1 node.go:158] NodeStageVolume: find device path /dev/xvdba -> /dev/xvdba
I0511 18:09:24.699995       1 node.go:193] NodeStageVolume: formatting /dev/xvdba and mounting at /csi/staging/ebs-vol0/rw-file-system-single-node-writer with fstype ext4
I0511 18:09:24.700008       1 mount_linux.go:274] Checking for issues with fsck on disk: /dev/xvdba
I0511 18:09:24.733185       1 mount_linux.go:287] `fsck` error fsck from util-linux 2.30.2
fsck.ext2: Bad magic number in super-block while trying to open /dev/xvdba
/dev/xvdba:
The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

I0511 18:09:24.733212       1 mount_linux.go:293] Attempting to mount disk: ext4 /dev/xvdba /csi/staging/ebs-vol0/rw-file-system-single-node-writer
I0511 18:09:24.733224       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t ext4 -o defaults /dev/xvdba /csi/staging/ebs-vol0/rw-file-system-single-node-writer])
E0511 18:09:24.741314       1 mount_linux.go:140] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t ext4 -o defaults /dev/xvdba /csi/staging/ebs-vol0/rw-file-system-single-node-writer
Output: mount: /csi/staging/ebs-vol0/rw-file-system-single-node-writer: wrong fs type, bad option, bad superblock on /dev/xvdba, missing codepage or helper program, or other error.

I0511 18:09:24.741350       1 mount_linux.go:346] Attempting to determine if disk "/dev/xvdba" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/xvdba])
I0511 18:09:24.751793       1 mount_linux.go:349] Output: "", err: exit status 2
I0511 18:09:24.751814       1 mount_linux.go:322] Disk "/dev/xvdba" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/xvdba]
I0511 18:09:26.094995       1 mount_linux.go:326] Disk successfully formatted (mkfs): ext4 - /dev/xvdba /csi/staging/ebs-vol0/rw-file-system-single-node-writer
I0511 18:09:26.095021       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t ext4 -o defaults /dev/xvdba /csi/staging/ebs-vol0/rw-file-system-single-node-writer])
I0511 18:09:26.103375       1 node.go:143] NodeStageVolume: volume="vol-024d9fe7b511b9be6" operation finished
I0511 18:09:26.103434       1 node.go:145] donedone
I0511 18:09:26.104369       1 node.go:279] NodePublishVolume: called with args {VolumeId:vol-024d9fe7b511b9be6 PublishContext:map[devicePath:/dev/xvdba] StagingTargetPath:/csi/staging/ebs-vol0/rw-file-system-single-node-writer TargetPath:/csi/per-alloc/a961ac5a-b76f-9a5a-6a52-07f21ed910aa/ebs-vol0/rw-file-system-single-node-writer VolumeCapability:mount:<> access_mode:<mode:SINGLE_NODE_WRITER >  Readonly:false Secrets:map[secretsauce:xyzzy] VolumeContext:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0511 18:09:26.104411       1 node.go:440] NodePublishVolume: creating dir /csi/per-alloc/a961ac5a-b76f-9a5a-6a52-07f21ed910aa/ebs-vol0/rw-file-system-single-node-writer
I0511 18:09:26.104424       1 node.go:450] NodePublishVolume: mounting /csi/staging/ebs-vol0/rw-file-system-single-node-writer at /csi/per-alloc/a961ac5a-b76f-9a5a-6a52-07f21ed910aa/ebs-vol0/rw-file-system-single-node-writer with option [bind] as fstype ext4
I0511 18:09:26.104434       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t ext4 -o bind /csi/staging/ebs-vol0/rw-file-system-single-node-writer /csi/per-alloc/a961ac5a-b76f-9a5a-6a52-07f21ed910aa/ebs-vol0/rw-file-system-single-node-writer])
I0511 18:09:26.106889       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t ext4 -o bind,remount /csi/staging/ebs-vol0/rw-file-system-single-node-writer /csi/per-alloc/a961ac5a-b76f-9a5a-6a52-07f21ed910aa/ebs-vol0/rw-file-system-single-node-writer])
I0511 18:09:42.485772       1 node.go:349] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0511 18:10:12.486556       1 node.go:349] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}

EBS controller unpublish:

I0511 18:14:07.456645       1 controller.go:275] ControllerUnpublishVolume: called with args {VolumeId:vol-024d9fe7b511b9be6 NodeId:i-087bb3ed4f5954a3a Secrets:map[secretsauce:xyzzy] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0511 18:14:12.723475       1 controller.go:298] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0511 18:14:20.126080       1 controller.go:292] ControllerUnpublishVolume: volume vol-024d9fe7b511b9be6 detached from node i-087bb3ed4f5954a3a
I0511 18:14:42.724676       1 controller.go:298] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}

CSI plugins can require credentials for some publishing and unpublishing workflow RPCs. Secrets are configured at the time of volume registration, stored in the volume struct, and then passed around as an opaque map by Nomad to the plugins.

cgbaker · 2020-05-11T20:18:59Z

should MountOptions be removed from the CSI Volume list stub structs in api and structs?

tgross · 2020-05-11T20:21:28Z

I missed it in the stub but we're redacting it in the full response only in the HTTP API: https://github.com/hashicorp/nomad/pull/7923/files#diff-91535e68265a2446b33edbefba3453d0. Will fix to include the stub response too.

github-actions · 2023-01-06T02:17:04Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

tgross added the theme/storage label May 11, 2020

tgross added this to the 0.11.2 milestone May 11, 2020

tgross force-pushed the csi_secrets branch 4 times, most recently from b752a52 to 9b1fc89 Compare May 11, 2020 19:27

tgross force-pushed the csi_secrets branch from 9b1fc89 to 4729737 Compare May 11, 2020 19:31

tgross marked this pull request as ready for review May 11, 2020 19:36

tgross requested review from langmartin and schmichael May 11, 2020 19:36

tgross mentioned this pull request May 11, 2020

support for nomad 0.11.1 API hashicorp/nomad-java-sdk#47

Merged

tgross force-pushed the csi_secrets branch from 4729737 to a0a7ba7 Compare May 11, 2020 19:56

csi: support Secrets parameter in CSI RPCs

c28c4c1

CSI plugins can require credentials for some publishing and unpublishing workflow RPCs. Secrets are configured at the time of volume registration, stored in the volume struct, and then passed around as an opaque map by Nomad to the plugins.

tgross force-pushed the csi_secrets branch from a0a7ba7 to c28c4c1 Compare May 11, 2020 19:59

langmartin approved these changes May 11, 2020

View reviewed changes

tgross force-pushed the csi_secrets branch from 474da34 to a801e1e Compare May 11, 2020 20:36

remove MountOptions from volume stub

f81dae3

tgross force-pushed the csi_secrets branch from a801e1e to f81dae3 Compare May 11, 2020 20:41

tgross merged commit a28f18e into master May 11, 2020

tgross deleted the csi_secrets branch May 11, 2020 21:12

This was referenced May 11, 2020

CSI implement secrets support #7668

Closed

CSI: support getting Secrets from Variables or Vault #7978

Open

github-actions bot locked as resolved and limited conversation to collaborators Jan 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

csi: support Secrets parameter in CSI RPCs #7923

csi: support Secrets parameter in CSI RPCs #7923

tgross commented May 11, 2020 •

edited

Loading

tgross commented May 11, 2020

cgbaker commented May 11, 2020

tgross commented May 11, 2020

github-actions bot commented Jan 6, 2023

csi: support Secrets parameter in CSI RPCs #7923

csi: support Secrets parameter in CSI RPCs #7923

Conversation

tgross commented May 11, 2020 • edited Loading

tgross commented May 11, 2020

cgbaker commented May 11, 2020

tgross commented May 11, 2020

github-actions bot commented Jan 6, 2023

tgross commented May 11, 2020 •

edited

Loading