Parameters on this idempotent request are inconsistent with parameters used in previous request

[rmvangun]

/kind bug

What happened?
I'm trying to create a new persistent volume and attach it to a pod. The data source for the pvc is a VolumeSnapshot that I have manually created in the cluster based on a snapshot that was created and shared from another account. I create a brand new PVC (new name, etc.) that points to this VolumeSnapshot datasource, and get the following:

Warning ProvisioningFailed 49s (x6 over 2m54s) ebs.csi.aws.com_ebs-csi-controller-5d5b7d7d8b-v55lb_e302255b-2caa-436f-8fa0-77cfda265bb4 failed to provision volume with StorageClass "my-ebs-sc": rpc error: code = AlreadyExists desc = Could not create volume "pvc-069f277c-0347-4dad-b87b-e922f662b22f": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)

Note that there are no persistent volumes that already exist, so I'm not sure what the AlreadyExists error is referring to.

What you expected to happen?
The volume is successfully provisioned

How to reproduce it (as minimally and precisely as possible)?

1. Create a new snapshot manually in AWS
2. Create a new VolumeSnapshotContents that references the EBS snapshot
3. Create the VolumeSnapshot and PersistentVolumeClaim with appropriate parameters (generally following https://www.stardog.com/labs/blog/eks-volume-snapshots/)

Anything else we need to know?:

• The original snapshot is encrypted with KMS in a different AWS account (I have set up correct IAM permissions)
• The new volume should also be created with a new KMS key

Environment

• Kubernetes version (use kubectl version): 1.21
• Driver version: 2.1.0

[nirmalaagash]

@rmvangun I tried to re-create the issue(created the snapshot manually from the same AWS account as the cluster nodes). From the steps in reproduce, it looks like you are using the storageclass which does not have the field encrypted:true (but your snapshot is encrypted)
Please follow the steps mentioned in the link to import your snapshot and create a new PV. Replace the storageclass.yaml used in step 2 with the below manifest.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp2
  "encrypted": "true"
allowVolumeExpansion: true

If problem still persists, please provide more information on how the snapshot from a different account is used.

[rmvangun]

This was related! It turns out that the volume was being momentarily created but then failed to encrypt, which eventually I determined from the logs. I had to take the extra step of sending EBS Cloudwatch Events to a Cloudwatch Log Group so I could get those extra KMS related error messages. There were multiple issues related to KMS permissions. In particular, I needed to add some KMS Grant related policies to the ebs-csi service account permissions.

This stuff is hard to debug! You really have to be streaming the logs from the ebs-csi-controller plus have these Cloudwatch Events set up to see what's going on. It's pretty dang nice once you do, I don't think I'll find myself in the dark again.

This was the post that really led me in the right direction...

https://aws.amazon.com/premiumsupport/knowledge-center/ebs-troubleshoot-kms-encrypted-snapshot/