Cant create raw block pvc from volume snapshot

[RaunakShah]

While helping @xing-yang test volume snapshots, we noticed that snapshots of raw block volumes cant be used as a source to create PVCs.

1. Create a raw block pvc

$ cat pvc-raw.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-raw
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: csi-hostpath-sc
  volumeMode: Block
  resources:
    requests:
      storage: 1Gi

2. Create a snapshot of the pvc

$ cat snapshot-raw.yaml 
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
  name: raw-pvc-snapshot
spec:
  volumeSnapshotClassName: csi-hostpath-snapclass
  source:
    persistentVolumeClaimName: pvc-raw

3. Create a pvc with snapshot above as the source

$ cat pvc-restore-raw.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-pvc-restore
spec:
  storageClassName: csi-hostpath-sc
  volumeMode: Block
  dataSource:
    name: raw-pvc-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

The restored PVC remains in the Pending state:

$ kubectl get pvc,volumesnapshot
NAME                                    STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
persistentvolumeclaim/pvc-raw           Bound     pvc-e67a6eaa-34ef-4841-a081-86900477b347   1Gi        RWO            csi-hostpath-sc   23s
persistentvolumeclaim/raw-pvc-restore   Pending                                                                        csi-hostpath-sc   1s

NAME                                                      READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS            SNAPSHOTCONTENT                                    CREATIONTIME   AGE
volumesnapshot.snapshot.storage.k8s.io/raw-pvc-snapshot   true         pvc-raw                             1Gi           csi-hostpath-snapclass   snapcontent-313de7ed-4c5b-4f7a-804a-e4e124938279   13s            13s

$ kubectl describe pvc 
...
Name:          raw-pvc-restore
Namespace:     default
StorageClass:  csi-hostpath-sc
Status:        Pending
...
Events:
  Type     Reason                Age                From                                                                                 Message
  ----     ------                ----               ----                                                                                 -------
  Warning  ProvisioningFailed    27s                hostpath.csi.k8s.io_csi-hostpath-provisioner-0_d61fffad-7655-456e-98b4-d98fc7f6d272  failed to provision volume with StorageClass "csi-hostpath-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Normal   ExternalProvisioning  14s (x3 over 37s)  persistentvolume-controller                                                          waiting for a volume to be created, either by external provisioner "hostpath.csi.k8s.io" or manually created by system administrator
  Normal   Provisioning          12s (x5 over 37s)  hostpath.csi.k8s.io_csi-hostpath-provisioner-0_d61fffad-7655-456e-98b4-d98fc7f6d272  External provisioner is provisioning volume for claim "default/raw-pvc-restore"
  Warning  ProvisioningFailed    12s (x4 over 26s)  hostpath.csi.k8s.io_csi-hostpath-provisioner-0_d61fffad-7655-456e-98b4-d98fc7f6d272  failed to provision volume with StorageClass "csi-hostpath-sc": rpc error: code = AlreadyExists desc = existing volume source snapshot id not matching

[RaunakShah]

/assign

[msau42]

cc @Jiawei0227

[Jiawei0227]

Interesting. Today in our snapshottable we dont have raw block tests. It seems we need to add it sooner than later

[Jiawei0227]

I have seen this error in the https://k8s-testgrid.appspot.com/sig-storage-csi-ci#1.18-on-master testing