forked from yard-turkey/rook
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: add design doc for Ceph COSI driver
The first draft for the design doc for ceph cosi driver with Rook Resolves rook#7843 Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
- Loading branch information
Showing
1 changed file
with
87 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| # Ceph COSI Driver Support | ||
|
|
||
| ## Targeted for v1.12 | ||
|
|
||
| ## Background | ||
|
|
||
| Container Object Storage Interface (COSI) is a specification for container orchestration frameworks to manage object storage. Even though there is no standard procotol defined for Object Store, it has flexibility to add support for all. The COSI spec abstracts common storage features such as create/delete buckets, grant/revoke access to buckets, attach/detach buckets, and more. COSI released v1alpha1 with Kubernetes 1.25. COSI is a new project and is not yet fully integrated by Kubernetes. | ||
| More details about COSI can be found [here](https://kubernetes.io/blog/2022/09/02/cosi-kubernetes-object-storage-management/) | ||
| It is projected that COSI will be the only supported object storage driver in the near future. In-tree drivers such as Ceph RGW will be replaced with their respective COSI drivers. | ||
|
|
||
| ## Current Status of Ceph COSI Driver | ||
|
|
||
| The [Ceph COSI driver](https://github.com/ceph/ceph-cosi) is currently in the pre-alpha status. It is currently being tested with latest COSI Spec and APIs with images: | ||
|
|
||
| - cosi-controller : gcr.io/k8s-staging-sig-storage/objectstorage-controller:v20230130-v0.1.1-12-geafd6fa | ||
| - cosi-sidecar : gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20230130-v0.1.0-24-gc0cf995 | ||
|
|
||
| The Ceph COSI driver supports RGW with ceph storage as backend with s3 protocol. Later on, it will be extended to support other protocols such as Swift and different backends for RGW via Zipper. | ||
|
|
||
| ## COSI Driver Deployment | ||
|
|
||
| The [COSI controller](https://github.com/kubernetes-sigs/container-object-storage-interface-controller) is deployed as container in the default namespace. The Ceph COSI driver is deployed as a statefulset with a single replica along with [COSI sidecar container](https://github.com/kubernetes-sigs/container-object-storage-interface-provisioner-sidecar). The Ceph COSI driver can be deployed in any namespace not along with the COSI controller. The Ceph COSI driver is deployed with a service account that has the following RBAC permissions: | ||
|
|
||
|
|
||
| ## Integration plan with Rook | ||
|
|
||
| The aim to support the v1alpha1 version of COSI in Rook v1.12. It will be extended to beta and release versions as appropriate. There should be option in `operator.yaml` to validate whether the COSI controller exists and another option at object store level to bring up the Ceph COSI driver. Each Ceph object store should have its own COSI driver. | ||
|
|
||
| ### Pre-requisites | ||
|
|
||
| - COSI CRDs should be installed in the cluster via following command | ||
|
|
||
| ```bash | ||
| kubectl apply -k github.com/kubernetes-sigs/container-object-storage-interface-api | ||
| ``` | ||
|
|
||
| - COSI controller should be deployed in the cluster via following command | ||
|
|
||
| ```bash | ||
| kubectl apply -k github.com/kubernetes-sigs/container-object-storage-interface-controller | ||
| ``` | ||
|
|
||
| - Create Service Account and RBAC for COSI driver | ||
|
|
||
| ```bash | ||
| kubectl create -f https://github.com/ceph/ceph-cosi/blob/master/resources/sa.yaml | ||
| kubectl create -f https://github.com/ceph/ceph-cosi/blob/master/resources/rbac.yaml | ||
| ``` | ||
|
|
||
| ### Changes in Operator.yaml | ||
|
|
||
| Below option validates whether the COSI controller is deployed in the cluster. If it is not deployed, it will log an warning message and when user tries to create object store woth cosi, it will reconcile until the controller is deployed. | ||
|
|
||
| ``` | ||
| cosiController: | ||
| enabled: true | ||
| ``` | ||
|
|
||
| ### Changes in ObjectStore.yaml | ||
|
|
||
| Following option will bring up the Ceph COSI driver for the RGW server. It will also create `CephObjectStoreUser` objectstorage-provisioner which provides credentials for Ceph COSI driver. | ||
| ``` | ||
| cephCosiDrvierSpec: | ||
| enabled: true | ||
| ``` | ||
|
|
||
| ### How Rook can improve COSI driver reliability | ||
|
|
||
| Rook can ensure that resources need for Ceph COSI driver such as Ceph object store is deployed and running in the cluster before creating requests for Bucket and its Access. Rook can also ensure that the COSI driver is deployed with the correct RBAC permissions, with the correct version of COSI controller. Rook should also prevent deletion of the ceph object Store if there are any buckets or bucket access in the cluster which already present with current codebase. Rook also need to provide secret containing the credentials for the ceph object store to the COSI driver. The secret need to updated if the ceph object store endpoint changes or credentials are changed. Rook need to bring up multiple drivers for multiple ceph object stores in the cluster. Rook also need to support update of COSI-Controller and COSI-Driver images independently | ||
|
|
||
| ### Coexistence of COSI and libbucket provisioner | ||
|
|
||
| Currently the ceph object store provisioned via Object Bucket Claim (OBC). They both can co exist and can even use same backend bucket from ceph storage. No deployment/configuration changes are required to support both. The libbucket probvisioner is deprecated and eventually will be replaced by COSI when it becomes more and more stable. The CRDs used by both are different hence there is no conflicts between them. | ||
|
|
||
| ### Ceph COSI Driver Requirements | ||
|
|
||
| - Ceph Object Store should be deployed and running in the cluster | ||
| - The credentials/endpoint for the ceph object store should be available by creating ceph object store user with proper permissions | ||
| - The COSI controller should be deployed in the cluster | ||
| - Rook can able to manage multiple ceph cosi drivers | ||
| - Rook should not modify cosi resource like Bucket, BucketAccess, BucketClaim, BucketAccessClass etc. | ||
|
|
||
| ### Rook Requirements | ||
|
|
||
| - Rook need to dynamically create/update the secret containing the credentials of the ceph object store for ceph COSI driver. | ||
| - User should not be required to deploy Rook differently when using COSI and OBC for ceph object store, expect the minimal changes in the `operator.yaml` and `objectstore.yaml`. | ||
| - When provisioning ceph COSI driver Rook must uniquely identify the driver/provisioner name so that multiple COSI drivers or multiple Rook instances within a Kubernetes cluster will not collide. |