[BUG] Restore action fails with OpenEBS cstor CSI Driver

[shlokc9]

Describe the bug
When running the Backup and Restore for MySQL app using CSI Snapshot functions at path ./examples/stable/mysql-csi-snapshot, following error is observed;

Warning ActionSetFailed Action: restore 14s Kanister Controller Failed to execute phase: v1alpha1.Phase{Name:"restoreCSISnapshot", State:"pending", Output:map[string]interface {}(nil)}: PersistentVolumeClaim "demo-cstor-vol5-restored" is invalid: spec.resources[storage]: Invalid value: "0": must be greater than zero

This bug is in continuation of the discussion that happened here

To Reproduce
Steps to reproduce the behavior:

1. Install OpenEBS cStor CSI Driver on the kubernetes environment
2. Execute the steps mentioned in the example as is (Only update the snapshotClass and storageClass fields in the blueprint before running it)
3. After running backup action, inspect the event logs by describing the backup actionset. Check for the restoreSize in outputArtifacts.
4. After running restore action, inspect the event logs by describing the restore actionset. Here is where you see the above error.

Expected behavior

1. restoreSize field in create actionset should be a non-zero and non-negative value (equal to PVC resource storage request)
2. Restore actionset should pass without any failure

Environment
Kubernetes Version/Provider: ...
Storage Provider: OpenEBS cStor CSI Driver
Cluster Size (#nodes): ...
Data Size: ...

[viveksinghggits]

#1184 (comment)

[shlokc9]

I tried installing openebs cStor driver on a kind cluster (Ubuntu machine), but, it looks like the pods are not transitioning to Running state since a long time. Followed the steps mentioned here: https://openebs.io/docs/user-guides/cstor

$ kubectl get pod -n openebs
NAME READY STATUS RESTARTS AGE
openebs-cstor-admission-server-5d7c9d7669-rssgp 0/1 ContainerCreating 0 86m
openebs-cstor-csi-controller-0 0/6 ContainerCreating 0 86m
openebs-cstor-csi-node-bm5ml 0/2 ContainerCreating 0 86m
openebs-cstor-cspc-operator-6bc968744c-bdldb 0/1 ContainerCreating 0 86m
openebs-cstor-cvc-operator-78c6c6c886-5bhhm 0/1 ContainerCreating 0 86m
openebs-localpv-provisioner-68cdbf6645-9nw5n 0/1 ContainerCreating 0 86m
openebs-ndm-fpshw 0/1 ContainerCreating 0 86m
openebs-ndm-operator-68c5465857-2j44t 0/1 ContainerCreating 0 86m

[shlokc9]

@oliverblaha Could you mention the platform that you used for installing iSCSI client? https://openebs.io/docs/user-guides/prerequisites

[github-actions]

This issue is marked as stale due to inactivity. Add a new comment to reactivate it.

[ihcsim]

This is a bug. We can't use args.RestoreSize in the restored PVC spec if it's 0.

[ihcsim]

We can reproduce this with a sample YAML like:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-0
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: gp2 // on eks
  resources:
    requests:
      storage: "0"

Then:

$ k apply -f pvc.yaml
The PersistentVolumeClaim "test-0" is invalid: spec.resources[storage]: Invalid value: "0": must be greater than zero

[github-actions]

This issue is marked as stale due to inactivity. Add a new comment to reactivate it.

[ihcsim]

@shlokchaudhari9 I'm going to submit a PR to update the RestoreCSISnapshot function to return an error, if restoreSize is empty or equal to zero. Per my testing below, Kubernetes requires the restoreSize in the restored PVC to be non-empty or greater than zero. If the cstor CSI driver returns a zero value, then it's a bug on their end.

Steps To Reproduce

Follow the instructions here to set up the CSI snapshot controller and the CSI hostpath driver:

Deploy this Nginx statefulset workload:

cat<<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  serviceName: nginx
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: csi-hostpath-sc
      resources:
        requests:
          storage: 1Gi
EOF

Confirm that the pods and their persistent volume claims are ready:

kubectl get po,pvc 

Deploy this VolumeSnapshot resource to snapshot the PVC:

cat<<EOF | kubectl apply -f -
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: nginx-sts
spec:
  volumeSnapshotClassName: csi-hostpath-snapclass
  source:
    persistentVolumeClaimName: www-nginx-0

Confirm that the VolumeSnapshot and VolumeSnapshotContent resources are ready:

kubectl get vs,vsc

Attempts to restore these PVCs will fail because Kubernetes require the restoreSize to be non-empty and greater than zero:

cat EOF<< | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restore-www-nginx-1
spec:
  storageClassName: csi-hostpath-sc
  dataSource:
    name: nginx-sts
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: "0"
EOF
The PersistentVolumeClaim "restore-www-nginx-1" is invalid: spec.resources[storage]: Invalid value: "0": must be greater than zero

cat EOF<< | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restore-www-nginx-1
spec:
  storageClassName: csi-hostpath-sc
  dataSource:
    name: nginx-sts
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
The PersistentVolumeClaim "restore-www-nginx-1" is invalid: spec.resources[storage]: Required value