docs: add design doc for Ceph COSI driver

The first draft for the design doc for ceph cosi driver with Rook

Resolves rook#7843

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

design/ceph/object/ceph-cosi-driver.md

@@ -0,0 +1,74 @@
+# Ceph COSI Driver Support
+
+## Targeted for v1.12
+
+## Background
+
+Container Object Storage Interface (COSI) is a set of specifications for container orchestration frameworks to manage object storage. Even though there is no standard procotol defined for Object Store, it has flexiblity to add support for all. The COSI spec abstracts common storage features such as create/delete buckets, grant/revoke access to buckets, attach/detach buckets, and more. It is currently at the alpha release. COSI is a new project and is not yet fully integrated by Kubernetes.
+More details about COSI can be found [here](https://kubernetes.io/blog/2022/09/02/cosi-kubernetes-object-storage-management/)
+It is projected that COSI will be the only supported object storage driver in the near feature. In-tree drivers such as Ceph RGW will be replaced with their respective COSI drivers.
+
+## Current Status of Ceph COSI Driver
+
+The [Ceph COSI driver](https://github.com/ceph/ceph-cosi) is currently in the pre-alpha status. It is currently being tested with latest COSI Spec and APIs with images:
+
+- cosi-controller : gcr.io/k8s-staging-sig-storage/objectstorage-controller:v20230130-v0.1.1-12-geafd6fa
+- cosi-sidecar : gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20230130-v0.1.0-24-gc0cf995
+
+The Ceph COSI driver supports RGW with ceph storage as backend with s3 protocol. Later on, it will be extended to support other protocols such as Swift and different backends for RGW via Zipper.
+
+## COSI Driver Deployment
+
+The [COSI controller](https://github.com/kubernetes-sigs/container-object-storage-interface-controller) is deployed as container in the default namespace. The Ceph COSI driver is deployed as a statefulset with a single replica along with [COSI sidecar container](https://github.com/kubernetes-sigs/container-object-storage-interface-provisioner-sidecar). The Ceph COSI driver can be deployed in any namespace not along with the COSI controller. The Ceph COSI driver is deployed with a service account that has the following RBAC permissions:
+
+```yaml
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: objectstorage-provisioner-role
+  labels:
+    app.kubernetes.io/part-of: container-object-storage-interface
+    app.kubernetes.io/component: driver-ceph
+    app.kubernetes.io/version: main
+    app.kubernetes.io/name: cosi-driver-ceph
+rules:
+- apiGroups: ["objectstorage.k8s.io"]
+  resources: ["buckets", "bucketaccesses", "bucketclaims", "bucketaccessclasses", "buckets/status", "bucketaccesses/status", "bucketclaims/status", "bucketaccessclasses/status"]
+  verbs: ["get", "list", "watch", "update", "create", "delete"]
+- apiGroups: ["coordination.k8s.io"]
+  resources: ["leases"]
+  verbs: ["get", "watch", "list", "delete", "update", "create"]
+- apiGroups: [""]
+  resources: ["secrets", "events"]
+  verbs: ["get", "delete", "update", "create"]
+```
+
+All the COSI CRDs are installed from <http://github.com/kubernetes-sigs/container-object-storage-interface-api>
+
+## Integration plan with Rook
+
+The aim to support alpha version of COSI in Rook v1.12 and depending on development of COSI, it will be extended to beta and GA versions in Rook Operaotr. There should be option in `Operator.yaml` to bring up the COSI controller and another option at object store level to bring up the ceph COSI driver.
+
+### How Rook can improve COSI driver reliability
+
+Rook can ensure that resources need for ceph cosi driver such as ceph object store is deployed and running in the cluster before creating requests for Bucket and its Access. Rook can also ensure that the COSI driver is deployed with the correct RBAC permissions, with the correct version of COSI controller. Rook should also prevent deletion of the ceph object Store if there are any buckets or bucket access in the cluster. Rook also need to provide secret containing the credentials for the ceph object store to the COSI driver. The secret need to updated if the ceph object store endpoint changes or credentials are changed. Rook need to bring up multiple drivers for  multiple ceph object stores in the cluster.
+
+### Coexistence of COSI and libbucket provisioner
+
+Currently the ceph object store provisioned via Object Bucket Claim (OBC). They both can co exist and can even use same backend bucket from ceph storage. No deployment/configuration changes are required to support both. The libbucket probvisioner is deprecated and eventually will be replaced by COSI when it becomes more and more stable. The CRDs used by both are different hence there is no conflicts between them.
+
+### Ceph COSI Driver Requirements
+
+- Ceph Object Store should be deployed and running in the cluster
+- The credentials/endpoint for the ceph object store should be available by creating ceph object store user with proper permissions
+- The COSI controller should be deployed in the cluster
+- Rook can able to manage multiple ceph cosi drivers
+- Rook should not modify cosi resource like Bucket, BucketAccess, BucketClaim, BucketAccessClass etc.
+
+### Rook Requirements
+
+- Rook must include all the RBAC permissions required by the COSI for the deployment.
+- Rook need to install all the COSI CRDs.
+- Rook need to dynamically create/update the secret containing the credentials of the ceph object store for ceph COSI driver.
+- User should not be required to deploy Rook differently when using COSI and OBC for ceph object store, expect the minimal changes in the `Operator.yaml` and `ObjectStore.yaml`.
+- When provisioning ceph COSI driver Rook must uniquely identify the driver/provisioner name so that multiple COSI drivers or multiple Rook instances within a (Kubernetes) cluster will not collide.