rook-ceph: Allow volume resizing

[surajssd]

This adds parameters to the storage class rook-ceph-block created by rook-ceph component. This will enable the PVs created by the storage class to be resized on the fly.

---

Upgrade process on existing clusters:

Once this is merged one can update the StorageClass by simply applying the rook-ceph component again by running following command:

lokoctl component apply rook-ceph

All the PVs created with older StorageClass lack following information:

apiVersion: v1
kind: PersistentVolume
...
spec:
  csi:
    controllerExpandSecretRef:
      name: rook-csi-rbd-provisioner
      namespace: rook

This can be patched, but it has to be done manually:

export namespace=default
export pvcname=populate-1
kubectl patch pv $(kubectl -n $namespace get pvc $pvcname -o jsonpath='{.spec.volumeName}') --patch '{"spec":{"csi":{"controllerExpandSecretRef":{"name":"rook-csi-rbd-provisioner","namespace":"rook"}}}}'

If not done then PVs don't resize and we might see errors like the one mentioned in the description of this issue: rook/rook#5670 (comment).

[surajssd]

To test this:

Deploy a Lokomotive cluster with at least 3 worker nodes that has multiple disks, I deployed c2.medium. Deploy rook and rook-ceph using following config:

component "rook" {}

component "rook-ceph" {
  storage_class {
    enable = true
    default = true
  }
}

Now once the components are deployed with the changed code in this PR, deploy this toy app which populates the volume with garbage data, which is good for testing.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: populate-1
  name: populate-1
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: populate-1
  template:
    metadata:
      labels:
        app: populate-1
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        runAsGroup: 65534
      containers:
      - image: surajd/fedora-networking
        name: fedora-networking
        command: ["/bin/sh"]
        args:
        - -c
        - "echo 'running command' && dd if=/dev/zero of=/data/populate-1.txt count=46080 bs=1048576 status=progress && echo 'sleeping' && sleep infinity"
        volumeMounts:
        - mountPath: "/data"
          name: data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: populate-1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: populate-1
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

Once the pod deployed using above config says sleeping in its logs, then edit the PVC using following command:

echo '
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: populate-1
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
' | kubectl apply -f -

Now wait for the status of the PVC to say that it has successfully changed the size and also it will reflect once you try to get the PVC:

$ kubectl get pvc -n default
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
populate-1   Bound    pvc-a1aa3e32-78cd-4268-b73d-e72dcec74d4a   100Gi      RWO            rook-ceph-block   40m

Now exec into the pod and run following command:

du -sh /data
dd if=/dev/zero of=/data/populate-$RANDOM.txt count=46080 bs=1048576 status=progress
du -sh /data

Now you will see that there are two file in /data dir. If you try to run the dd command again it will error out.

[surajssd]

Corresponding docs issue raised upstream: rook/rook#5670

[iaguis]

lgtm

[ipochi]

Great work @surajssd 🎉

Tested.

$ du -sh /data
46G	/data

Resized PVC to 100GB

Executed dd command
$ du -sh /data 91G /data
Tried dd again , went as far as 100GB and then error of no space left.

$ dd if=/dev/zero of=/data/populate-$RANDOM.txt count=46080 bs=1048576 status=progress
9912188928 bytes (9.9 GB, 9.2 GiB) copied, 23 s, 432 MB/s
dd: error writing '/data/populate-6670.txt': No space left on device

LGTM