doc: add design doc for RBD QoS

docs/design/proposals/rbd-qos.md

@@ -0,0 +1,271 @@
+# Design Doc for RBD QoS using cgroup v2
+
+## Introduction
+
+The RBD QoS (Quality of Service) design aims to address the issue of IO noisy
+neighbor problems encountered in early Ceph deployments catering to OpenStack
+environments. These problems were effectively managed by implementing QEMU
+throttling at the virtio-blk/scsi level. To further enhance this,
+capacity-based IOPS were introduced, providing a more dynamic experience
+similar to public cloud environments.
+
+The challenge arises in virtual environments, where a noisy neighbor can lead
+to performance degradation for other instances sharing the same resources.
+Although it's uncommon to observe noisy neighbor issues in Kubernetes
+environments backed by Ceph storage, the possibility exists. The existing QoS
+support with rbd-nbd doesn't apply to krbd, and as rbd-nbd isn't suitable for
+container production workloads, a solution is needed for krbd.
+
+To mitigate resource starvation issues, setting QoS at the device level through
+cgroup v2 when enabled becomes crucial. This approach guarantees that I/O
+capacity isn't overcommitted and is fairly distributed among workloads.
+
+## Dependency
+
+* cgroup v2 must be enabled on the Node
+* We might have Kubernetes dependency as well
+* Container runtime dependency that supports cgroupv2
+
+## Manual steps for implementing RBD QoS in a Kubernetes Cluster
+
+```bash
+[$] ssh root@node1
+sh-4.4# chroot /host
+sh-5.1# cat /proc/partitions
+major minor  #blocks  name
+
+ 259        0  125829120 nvme0n1
+ 259        1       1024 nvme0n1p1
+ 259        2     130048 nvme0n1p2
+ 259        3     393216 nvme0n1p3
+ 259        4  125303791 nvme0n1p4
+ 259        6   52428800 nvme2n1
+   7        0  536870912 loop0
+ 259        5  536870912 nvme1n1
+ 252        0   52428800 rbd0
+sh-5.1#
+```
+
+Once the rbd device is mapped on the node we get the device's major and minor
+number we need to set the io limit on the device but we need to find the right
+cgroup file where we need to set the limit
+
+Kubernetes/Openshift creates a custom cgroup hierarchy for the pods it created
+but start is `/sys/fs/cgroup`  folder
+
+```bash
+sh-5.1# cd /sys/fs/cgroup/
+sh-5.1# ls
+cgroup.controllers cgroup.subtree_control cpuset.mems.effective  io.stat   memory.reclaim   sys-kernel-debug.mount
+cgroup.max.depth cgroup.threads  dev-hugepages.mount    kubepods.slice  memory.stat   sys-kernel-tracing.mount
+cgroup.max.descendants cpu.pressure  dev-mqueue.mount       machine.slice  misc.capacity   system.slice
+cgroup.procs  cpu.stat  init.scope        memory.numa_stat  sys-fs-fuse-connections.mount user.slice
+cgroup.stat  cpuset.cpus.effective io.pressure        memory.pressure  sys-kernel-config.mount
+```
+
+`kubepods.slice` is the starting point and it contains multiple slices
+
+```bash
+sh-5.1# cd kubepods.slice
+sh-5.1# ls
+cgroup.controllers cpuset.cpus    hugetlb.2MB.rsvd.max       memory.pressure
+cgroup.events  cpuset.cpus.effective   io.bfq.weight        memory.reclaim
+cgroup.freeze  cpuset.cpus.partition   io.latency        memory.stat
+cgroup.kill  cpuset.mems    io.max        memory.swap.current
+cgroup.max.depth cpuset.mems.effective   io.pressure        memory.swap.events
+cgroup.max.descendants hugetlb.1GB.current   io.stat        memory.swap.high
+cgroup.procs  hugetlb.1GB.events   kubepods-besteffort.slice      memory.swap.max
+cgroup.stat  hugetlb.1GB.events.local  kubepods-burstable.slice      memory.zswap.current
+cgroup.subtree_control hugetlb.1GB.max    kubepods-pod2b38830b_c2d6_4528_8935_b1c08511b1e3.slice  memory.zswap.max
+cgroup.threads  hugetlb.1GB.numa_stat   memory.current       misc.current
+cgroup.type  hugetlb.1GB.rsvd.current  memory.events        misc.max
+cpu.idle  hugetlb.1GB.rsvd.max   memory.events.local       pids.current
+cpu.max   hugetlb.2MB.current   memory.high        pids.events
+cpu.max.burst  hugetlb.2MB.events   memory.low        pids.max
+cpu.pressure  hugetlb.2MB.events.local  memory.max        rdma.current
+cpu.stat  hugetlb.2MB.max    memory.min        rdma.max
+cpu.weight  hugetlb.2MB.numa_stat   memory.numa_stat
+cpu.weight.nice  hugetlb.2MB.rsvd.current  memory.oom.group
+```
+
+Based on the QoS of the pod, either our application pod will end up in the
+above `kubepods-besteffort.slice` or `kubepods-burstable.slice` or
+`kubepods.slice` (Guaranteed QoS) cgroup. The 3 QoS classes are defined
+[here](https://kubernetes.io/docs/concepts/workloads/pods/pod-QoS/#quality-of-service-classes)
+
+To identify the right cgroup file, we need pod UUID and container UUID from the
+`pod yaml` output
+
+```bash
+[$]kubectl get po csi-rbd-demo-pod -oyaml |grep uid
+  uid: cdf7b785-4eb7-44f7-99cc-ef53890f4dfd
+[$]kubectl get po csi-rbd-demo-pod -oyaml |grep -i containerID
+  - containerID: cri-o://77e57fbbc0f0630f41f9f154f4b5fe368b6dcf7bef7dcd75a9c4b56676f10bc9
+[$]kubectl get po csi-rbd-demo-pod -oyaml |grep -i qosClass
+  qosClass: BestEffort
+```
+
+Now check in the `kubepods-besteffort.slice` and identify the right path using
+pod UID and container UID
+
+Before that check `io.max` on the application pod and see if there is any limit
+
+```bash
+[$]kubectl exec -it csi-rbd-demo-pod -- sh
+sh-4.4# cat /sys/fs/cgroup/io.max
+sh-4.4#
+```
+
+Come back to the Node and find the right cgroup scope
+
+```bash
+sh-5.1# cd kubepods-besteffort.slice/kubepods-besteffort-podcdf7b785_4eb7_44f7_99cc_ef53890f4dfd.slice/crio-77e57fbbc0f0630f41f9f154f4b5fe368b6dcf7bef7dcd75a9c4b56676f10bc9.scope/
+
+
+sh-5.1# echo "252:0 wbps=1048576" > io.max
+sh-5.1# cat io.max
+252:0 rbps=max wbps=1048576 riops=max wiops=max
+```
+
+Now go back to the application pod and check if we have the right limit set
+
+```bash
+[$]kubectl exec -it csi-rbd-demo-pod -- sh
+sh-4.4# cat /sys/fs/cgroup/io.max
+252:0 rbps=max wbps=1048576 riops=max wiops=max
+sh-4.4#
+```
+
+Note:- We can only support the QoS that cgroup v2 io controller supports, this
+means that cumulative read+write QoS limits won't be supported.
+
+Below are the configurations that will be supported
+
+|  Parameter     |  Description     |
+|  ---  |  ---  |
+|  MaxReadIOPS     | Max read IO operations per second      |
+|  MaxWriteIOPS     | Max write IO operations per second      |
+|  MaxReadBytesPerSecond     |  Max read bytes per second     |
+|  MaxWriteBytesPerSecond     |  Max write bytes per second     |
+
+## Different approaches
+
+The above solution can be implemented using 3 different approaches.
+
+### 1. QoS using new parameters in RBD StorageClass
+
+```yaml
+---
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+   name: csi-rbd-sc
+provisioner: rbd.csi.ceph.com
+parameters:
+  MaxReadIOPS: ""
+  MaxWriteIOPS: ""
+  MaxReadBytesPerSecond: ""
+  MaxWriteBytesPerSecond: ""
+```
+
+#### Implementation for StorageClass QoS
+
+1. Create new storageClass with new parameters for QoS
+1. Modify CSIDriver object to pass pod details to the NodePublishVolume CSI
+   procedure
+1. During NodePublishVolume CSI procedure
+   * Retrieve the QoS configuration from the volumeContext in NodePublishRequest
+   * Identify the rbd device using the NodeStageVolumePath
+   * Get the pod UUID from the NodeStageVolume
+   * Set io.max file in all the containers in the pod
+
+#### Drawbacks of StorageClass QoS
+
+1. No way to update the QoS at runtime
+1. Need to take a backup and restore to New QoS StorageClass
+1. Delete and Recreate the PV object
+
+### 2. QoS using parameters in VolumeAttributeClass
+
+```yaml
+apiVersion: storage.k8s.io/v1alpha1
+kind: VolumeAttributesClass
+metadata:
+  name: silver
+parameters:
+  MaxReadIOPS: ""
+  MaxWriteIOPS: ""
+  MaxReadBytesPerSecond: ""
+  MaxWriteBytesPerSecond: ""
+```
+
+VolumeAttributesClassName is a new parameter in the PVC object the user can
+choose from and this can also be updated or removed later.
+
+This new VolumeAttributeClass is designed to keep storage that supports setting
+QoS at the storage level which means setting some configuration at the storage
+(like QoS for nbd)
+
+#### Implementation of VolumeAttributeClass QoS
+
+1. Modify CSIDriver object to pass pod details to the NodePublishVolume CSI
+   procedure
+1. Add support in Ceph-CSI to expose ModifyVolume CSI procedure
+1. Ceph-CSI will store QoS in the rbd image metadata
+1. During NodeStage operation retrieve the image metadata and store it in
+   stagingPath
+1. Whenever a new pod comes in apply the QoS
+
+#### Drawbacks of VolumeAttributeClass QoS
+
+One problem with above is all application need to be scaled downed and scaled
+up to get the new QoS value even though its changed in the PVC object, this is
+sometime impossible as it will have downtime.
+
+### 3. QoS using parameters in VolumeAttributeClass with NodePublish Secret
+
+1. Modify CSIDriver object to pass pod details to the NodePublishVolume CSI
+   procedure
+1. Add support in Ceph-CSI to expose ModifyVolume CSI procedure
+1. Ceph-CSI will store QoS in the rbd image metadata
+1. During NodePublishVolume operation retrieve the QoS from image metadata
+1. Whenever a new pod comes in apply the QoS
+
+This solution addresses the aforementioned issue, but it requires a secret to
+communicate with the ceph cluster. Therefore, we must create a new
+PublishSecret for the storageClass, which may be beneficial when Kubernetes
+eventually enables Node operations.
+
+Both options 2 and 3 are contingent upon changes to the CSI spec and Kubernetes
+support. Additionally,
+[VolumeAttributeClass](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/3751-volume-attributes-class/README.md)
+is currently being developed within the Kubernetes realm and will initially be
+in the Alpha stage. Consequently, it will be disabled by default.
+
+#### Advantages of QoS using VolumeAttributeClass
+
+1. No Restore/Clone operation is required to change the QoS
+1. Easily QoS can be changed for existing PVC only with second approach not
+   with third as it needs new secret.
+
+### Hybrid Approach
+
+Considering the advantages and drawbacks, we can use StorageClass and
+VolumeAttributeClass to support QoS, with VolumeAttributeClass taking
+precedence over StorageClass. This approach offers a flexible solution that
+accounts for dynamic changes while addressing the challenges of existing
+approaches.
+
+### References
+
+Some of the useful links that helped me to understand cgroup v2 and how to set
+QoS on the device.
+
+* [Kubernetes cgroup v2
+  Architecture](https://kubernetes.io/docs/concepts/architecture/cgroups/)
+* [cgroup v2 kernel doc](https://docs.kernel.org/admin-guide/cgroup-v2.html)
+* [ceph RBD QoS tracker](https://tracker.ceph.com/issues/36191)
+* [cgroup v2 io
+  controller](https://facebookmicrosites.github.io/cgroup2/docs/io-controller.html)
+* [Kubernetes IOPS
+  issue](https://github.com/kubernetes/kubernetes/issues/92287)