docs: add design doc for Ceph COSI driver

The first draft for the design doc for ceph cosi driver with Rook

Resolves rook#7843

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

design/ceph/object/ceph-cosi-driver.md

@@ -0,0 +1,83 @@
+# Ceph COSI Driver Support
+
+## Targeted for v1.12
+
+## Background
+
+Container Object Storage Interface (COSI) is a specification for container orchestration frameworks to manage object storage. Even though there is no standard procotol defined for Object Store, it has flexibility to add support for all. The COSI spec abstracts common storage features such as create/delete buckets, grant/revoke access to buckets, attach/detach buckets, and more. COSI released v1alpha1 with Kubernetes 1.25. COSI is a new project and is not yet fully integrated by Kubernetes.
+More details about COSI can be found [here](https://kubernetes.io/blog/2022/09/02/cosi-kubernetes-object-storage-management/)
+It is projected that COSI will be the only supported object storage driver in the near future. In-tree drivers such as Ceph RGW will be replaced with their respective COSI drivers.
+
+
+## COSI Driver Deployment
+
+The [COSI controller](https://github.com/kubernetes-sigs/container-object-storage-interface-controller) is deployed as container in the default namespace. The Ceph COSI driver is deployed as a statefulset with a single replica along with [COSI sidecar container](https://github.com/kubernetes-sigs/container-object-storage-interface-provisioner-sidecar). The Ceph COSI driver can be deployed in any namespace not along with the COSI controller. The Ceph COSI driver is deployed with a service account that has the following RBAC permissions:
+
+
+## Integration plan with Rook
+
+The aim to support the v1alpha1 version of COSI in Rook v1.12. It will be extended to beta and release versions as appropriate. There should be option in `operator.yaml` to validate whether the COSI controller exists and another option at object store level to bring up the Ceph COSI driver. Each Ceph object store should have its own COSI driver.
+
+### Pre-requisites
+
+- COSI CRDs should be installed in the cluster via following command
+
+```bash
+kubectl apply -k github.com/kubernetes-sigs/container-object-storage-interface-api
+```
+
+- COSI controller should be deployed in the cluster via following command
+
+```bash
+kubectl apply -k github.com/kubernetes-sigs/container-object-storage-interface-controller
+```
+
+- Create Service Account and RBAC for COSI driver
+
+```bash
+kubectl create -f https://github.com/ceph/ceph-cosi/blob/master/resources/sa.yaml
+kubectl create -f https://github.com/ceph/ceph-cosi/blob/master/resources/rbac.yaml
+```
+
+### Changes in Operator.yaml
+
+Below option validates whether the COSI controller is deployed in the cluster. If it is not deployed, it will log an warning message and when user tries to create object store with cosi, it will reconcile until the controller is deployed.
+
+```
+  env:
+  - name: COSI_CONTROLLER_ENABLED
+    value: "true"
+```
+
+### Changes in CephObjectStore CRD
+
+Following option will bring up the Ceph COSI driver for the RGW server. It will also create `CephObjectStoreUser`  objectstorage-provisioner which provides credentials for Ceph COSI driver.
+```
+  cephCosiDrvierSpec:
+    enabled: true
+```
+
+### How Rook can improve COSI driver reliability
+
+Rook can ensure that resources need for Ceph COSI driver such as Ceph object store is deployed and running in the cluster before creating requests for Bucket and its Access. Rook can also ensure that the COSI driver is deployed with the correct RBAC permissions, with the correct version of COSI controller. Rook should also prevent deletion of the ceph object Store if there are any buckets or bucket access in the cluster which already present with current codebase. Rook also need to provide secret containing the credentials for the ceph object store to the COSI driver. The secret need to updated if the ceph object store endpoint changes or credentials are changed. Rook need to bring up multiple drivers for multiple ceph object stores in the cluster. Rook also need to support update of COSI-Controller and COSI-Driver images independently
+
+### Coexistence of COSI and libbucket provisioner
+
+Currently the ceph object store provisioned via Object Bucket Claim (OBC). They both can co exist and can even use same backend bucket from ceph storage. No deployment/configuration changes are required to support both. The libbucket probvisioner is deprecated and eventually will be replaced by COSI when it becomes more and more stable. The CRDs used by both are different hence there is no conflicts between them.
+
+#### Transition from libbucket provisioner to COSI
+
+This applied to OBC with reclaim policy is `Retain`. First the user need to create a COSI bucket pointing to the backend bucket. Then user can create `bucket access class` and `bucket access` using the `COSI bucket`. Now the update applications credentials with bucket access credentials, finally the user need to delete the OBC.
+### Ceph COSI Driver Requirements
+
+- Ceph Object Store should be deployed and running in the cluster
+- The credentials/endpoint for the ceph object store should be available by creating ceph object store user with proper permissions
+- The COSI controller should be deployed in the cluster
+- Rook can able to manage multiple ceph cosi drivers
+- Rook should not modify cosi resource like Bucket, BucketAccess, BucketClaim, BucketAccessClass etc.
+
+### Rook Requirements
+
+- Rook need to dynamically create/update the secret containing the credentials of the ceph object store for ceph COSI driver.
+- User should not be required to deploy Rook differently when using COSI and OBC for ceph object store, expect the minimal changes in the `operator.yaml` and `objectstore.yaml`.
+- When provisioning ceph COSI driver Rook must uniquely identify the driver name so that multiple COSI drivers or multiple Rook instances within a Kubernetes cluster will not collide.