nomad snapshot create does use persisted secrets

[ggriffiths]

Nomad version

Nomad v1.1.0 (2678c3604bc9530014208bc167415e167fd440fc)

Operating system and Environment details

CentOS-7

Issue

Nomad volume create does not seem to persist any values from the secrets block. The secrets are passed to the CSI Driver, as I'm able to use auth-token in the Portworx CSI Driver create call, however when creating a snapshot, I'm not seeing these secrets propagated to the snapshot create call. I'm not sure if this is by design or not. If this is by design, is there a way to have secrets passed to nomad volume snapshot create/delete?

The nomad volume status --json command shows null secrets:

[root@grantdev nomadspecs]# nomad volume status --json pxvol2 | grep Secrets
    "Secrets": null,

Reproduction steps

1. Create a typical volume with secrets

id           = "pxvol2"
name         = "database2"
type         = "csi"
plugin_id    = "portworx"
capacity_min = "5G"
capacity_max = "5G"

capability {
  access_mode     = "single-node-reader-only"
  attachment_mode = "file-system"
}

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

secrets {
  auth-token = "eyJhbGciOiJIUzI1NiIsIn..."
  test = "123"
}

parameters {
  abc = "def"
}

2. Check for secrets in volume, it will be null:

# nomad volume status --json pxvol2 | grep Secrets
    "Secrets": null,

3. Attempt to create volume snapshot with secrets from volume create, command will fail if it requires a secret:

# nomad volume snapshot create pxvol1 snap6
Error snapshotting volume: Unexpected response code: 500 (1 error occurred:
	* could not create snapshot: controller create snapshot: rpc error: controller create snapshot: CSI.ControllerCreateSnapshot: controller plugin returned an internal error, check the plugin allocation logs for more information: rpc error: code = Internal desc = Failed to create snapshot: rpc error: code = PermissionDenied desc = Access denied without authentication token

)

Expected Result

Secrets should be persisted during volume create

Actual Result

Secrets are not persisted during volume create

Job file (if appropriate)

n/a

Nomad Server logs (if appropriate)

n/a

Nomad Client logs (if appropriate)

n/a

[tgross]

Hi @ggriffiths. Thanks for opening this issue!

Just FYI that secrets are intentionally redacted from any query request, so that nomad volume status doesn't return them isn't unexpected. I took a quick pass over the code and it looks like the problem is the snapshot create just isn't getting the secrets parameters from the volume. The API endpoint assumes the command passed them in, whereas the command assumes the API endpoint will get them out of the persisted volume.

[ggriffiths]

Of course! That makes sense. If the fix is straightforward enough, I'd be happy to send a PR!

Looks like from a first glance, this line:

nomad/nomad/csi_endpoint.go

Line 1128 in 97af3e9

Secrets: snap.Secrets,

might need to be vol.Secrets?

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.