From 668af1f9d0b57b85f195c727a772739fe6a7d49c Mon Sep 17 00:00:00 2001 From: Rakshith R Date: Mon, 26 Jun 2023 11:16:24 +0530 Subject: [PATCH] doc: add steps for upgrading from 3.8 to 3.9 This commit adds steps for upgrading from 3.8 to 3.9. Signed-off-by: Rakshith R --- docs/ceph-csi-upgrade.md | 123 +++++++++++++++++++++++++-------------- 1 file changed, 79 insertions(+), 44 deletions(-) diff --git a/docs/ceph-csi-upgrade.md b/docs/ceph-csi-upgrade.md index 491e8aefd7af..f892807e6fdd 100644 --- a/docs/ceph-csi-upgrade.md +++ b/docs/ceph-csi-upgrade.md @@ -1,40 +1,42 @@ # Ceph-csi Upgrade - [Ceph-csi Upgrade](#ceph-csi-upgrade) - - [Pre-upgrade considerations](#pre-upgrade-considerations) - - [Snapshot-controller and snapshot crd](#snapshot-controller-and-snapshot-crd) - - [Snapshot API version support matrix](#snapshot-api-version-support-matrix) - - [Upgrading from v3.2 to v3.3](#upgrading-from-v32-to-v33) - - [Upgrading from v3.3 to v3.4](#upgrading-from-v33-to-v34) - - [Upgrading from v3.4 to v3.5](#upgrading-from-v34-to-v35) - - [Upgrading from v3.5 to v3.6](#upgrading-from-v35-to-v36) - - [Upgrading from v3.6 to v3.7](#upgrading-from-v36-to-v37) - - [Upgrading from v3.7 to v3.8](#upgrading-from-v37-to-v38) - - [Upgrading CephFS](#upgrading-cephfs) - - [1. Upgrade CephFS Provisioner resources](#1-upgrade-cephfs-provisioner-resources) - - [1.1 Update the CephFS Provisioner RBAC](#11-update-the-cephfs-provisioner-rbac) - - [1.2 Update the CephFS Provisioner deployment](#12-update-the-cephfs-provisioner-deployment) - - [2. Upgrade CephFS Nodeplugin resources](#2-upgrade-cephfs-nodeplugin-resources) - - [2.1 Update the CephFS Nodeplugin RBAC](#21-update-the-cephfs-nodeplugin-rbac) - - [2.2 Update the CephFS Nodeplugin daemonset](#22-update-the-cephfs-nodeplugin-daemonset) - - [2.3 Manual deletion of CephFS Nodeplugin daemonset pods](#23-manual-deletion-of-cephfs-nodeplugin-daemonset-pods) - - [Delete removed CephFS PSP, Role and RoleBinding](#delete-removed-cephfs-psp-role-and-rolebinding) - - [Upgrading RBD](#upgrading-rbd) - - [3. Upgrade RBD Provisioner resources](#3-upgrade-rbd-provisioner-resources) - - [3.1 Update the RBD Provisioner RBAC](#31-update-the-rbd-provisioner-rbac) - - [3.2 Update the RBD Provisioner deployment](#32-update-the-rbd-provisioner-deployment) - - [4. Upgrade RBD Nodeplugin resources](#4-upgrade-rbd-nodeplugin-resources) - - [4.1 Update the RBD Nodeplugin RBAC](#41-update-the-rbd-nodeplugin-rbac) - - [4.2 Update the RBD Nodeplugin daemonset](#42-update-the-rbd-nodeplugin-daemonset) - - [Delete removed RBD PSP, Role and RoleBinding](#delete-removed-rbd-psp-role-and-rolebinding) - - [Upgrading NFS](#upgrading-nfs) - - [5. Upgrade NFS Provisioner resources](#5-upgrade-nfs-provisioner-resources) - - [5.1 Update the NFS Provisioner RBAC](#51-update-the-nfs-provisioner-rbac) - - [5.2 Update the NFS Provisioner deployment](#52-update-the-nfs-provisioner-deployment) - - [6. Upgrade NFS Nodeplugin resources](#6-upgrade-nfs-nodeplugin-resources) - - [6.1 Update the NFS Nodeplugin RBAC](#61-update-the-nfs-nodeplugin-rbac) - - [6.2 Update the NFS Nodeplugin daemonset](#62-update-the-nfs-nodeplugin-daemonset) - - [CSI Sidecar containers consideration](#csi-sidecar-containers-consideration) + - [Pre-upgrade considerations](#pre-upgrade-considerations) + - [Snapshot-controller and snapshot crd](#snapshot-controller-and-snapshot-crd) + - [Snapshot API version support matrix](#snapshot-api-version-support-matrix) + - [Upgrading from v3.2 to v3.3](#upgrading-from-v32-to-v33) + - [Upgrading from v3.3 to v3.4](#upgrading-from-v33-to-v34) + - [Upgrading from v3.4 to v3.5](#upgrading-from-v34-to-v35) + - [Upgrading from v3.5 to v3.6](#upgrading-from-v35-to-v36) + - [Upgrading from v3.6 to v3.7](#upgrading-from-v36-to-v37) + - [Upgrading from v3.7 to v3.8](#upgrading-from-v37-to-v38) + - [Upgrading from v3.8 to v3.9](#upgrading-from-v38-to-v39) + - [Upgrading CephFS](#upgrading-cephfs) + - [1. Upgrade CephFS Provisioner resources](#1-upgrade-cephfs-provisioner-resources) + - [1.1 Update the CephFS Provisioner RBAC](#11-update-the-cephfs-provisioner-rbac) + - [1.2 Update the CephFS Provisioner deployment](#12-update-the-cephfs-provisioner-deployment) + - [2. Upgrade CephFS Nodeplugin resources](#2-upgrade-cephfs-nodeplugin-resources) + - [2.1 Update the CephFS Nodeplugin RBAC](#21-update-the-cephfs-nodeplugin-rbac) + - [2.2 Update the CephFS Nodeplugin daemonset](#22-update-the-cephfs-nodeplugin-daemonset) + - [2.3 Manual deletion of CephFS Nodeplugin daemonset pods](#23-manual-deletion-of-cephfs-nodeplugin-daemonset-pods) + - [2.4 Modifying MountOptions in Storageclass and PersistentVolumes](#24-modifying-mountoptions-in-storageclass-and-persistentvolumes) + - [Delete removed CephFS PSP, Role and RoleBinding](#delete-removed-cephfs-psp-role-and-rolebinding) + - [Upgrading RBD](#upgrading-rbd) + - [3. Upgrade RBD Provisioner resources](#3-upgrade-rbd-provisioner-resources) + - [3.1 Update the RBD Provisioner RBAC](#31-update-the-rbd-provisioner-rbac) + - [3.2 Update the RBD Provisioner deployment](#32-update-the-rbd-provisioner-deployment) + - [4. Upgrade RBD Nodeplugin resources](#4-upgrade-rbd-nodeplugin-resources) + - [4.1 Update the RBD Nodeplugin RBAC](#41-update-the-rbd-nodeplugin-rbac) + - [4.2 Update the RBD Nodeplugin daemonset](#42-update-the-rbd-nodeplugin-daemonset) + - [Delete removed RBD PSP, Role and RoleBinding](#delete-removed-rbd-psp-role-and-rolebinding) + - [Upgrading NFS](#upgrading-nfs) + - [5. Upgrade NFS Provisioner resources](#5-upgrade-nfs-provisioner-resources) + - [5.1 Update the NFS Provisioner RBAC](#51-update-the-nfs-provisioner-rbac) + - [5.2 Update the NFS Provisioner deployment](#52-update-the-nfs-provisioner-deployment) + - [6. Upgrade NFS Nodeplugin resources](#6-upgrade-nfs-nodeplugin-resources) + - [6.1 Update the NFS Nodeplugin RBAC](#61-update-the-nfs-nodeplugin-rbac) + - [6.2 Update the NFS Nodeplugin daemonset](#62-update-the-nfs-nodeplugin-daemonset) + - [CSI Sidecar containers consideration](#csi-sidecar-containers-consideration) ## Pre-upgrade considerations @@ -56,7 +58,7 @@ To avoid this issue in future upgrades, we recommend that you do not use the fuse client as of now. This guide will walk you through the steps to upgrade the software in a cluster -from v3.7 to v3.8 +from v3.8 to v3.9 ### Snapshot-controller and snapshot crd @@ -100,6 +102,11 @@ to upgrade from cephcsi v3.6 to v3.7 ## Upgrading from v3.7 to v3.8 +Refer [upgrade-from-v3.7-v3.8](https://github.com/ceph/ceph-csi/blob/v3.8.0/docs/ceph-csi-upgrade.md) +to upgrade from cephcsi v3.7 to v3.8 + +## Upgrading from v3.8 to v3.9 + **Ceph-csi releases from devel are expressly unsupported.** It is strongly recommended that you use [official releases](https://github.com/ceph/ceph-csi/releases) of Ceph-csi. Unreleased @@ -108,15 +115,15 @@ that will not be supported in the official releases. Builds from the devel branch can have functionality changed and even removed at any time without compatibility support and without prior notice. -**Also, we do not recommend any direct upgrades to 3.8 except from 3.7 to 3.8.** -For example, upgrading from 3.6 to 3.8 is not recommended. +**Also, we do not recommend any direct upgrades to 3.9 except from 3.8 to 3.9.** +For example, upgrading from 3.7 to 3.9 is not recommended. -git checkout v3.8.0 tag +git checkout v3.9.0 tag ```bash git clone https://github.com/ceph/ceph-csi.git cd ./ceph-csi -git checkout v3.8.0 +git checkout v3.9.0 ``` ```console @@ -127,6 +134,9 @@ Warning: kubectl apply should be used on resource created by either kubectl crea ### Upgrading CephFS +If existing cephfs storageclasses' `MountOptions` are set, please refer to +[modifying mount options](#24-modifying-mountoptions-in-storageclass-and-persistentvolumes) +section. Upgrading cephfs csi includes upgrade of cephfs driver and as well as kubernetes sidecar containers and also the permissions required for the kubernetes sidecar containers, lets upgrade the things one by one @@ -232,12 +242,37 @@ For each node: - Drain your application pods from the node - Delete the CSI driver pods on the node - - The pods to delete will be named with a csi-cephfsplugin prefix and have a + - The pods to delete will be named with a csi-cephfsplugin prefix and have a random suffix on each node. However, no need to delete the provisioner pods: csi-cephfsplugin-provisioner-* . - - The pod deletion causes the pods to be restarted and updated automatically + - The pod deletion causes the pods to be restarted and updated automatically on the node. +##### 2.4 Modifying MountOptions in Storageclass and PersistentVolumes + +CephCSI, starting from release v3.9.0, will pass the options specified in the +StorageClass's `MountOptions` during both `NodeStage` (kernel cephfs or +ceph-fuse mount operation) and `NodePublish` (bind mount) operations. +Therefore, only common options that is acceptable during both the above +described operation needs to be set in StorageClass's `MountOptions`. +If invalid mount options are set in StorageClass's `MountOptions` +such as `"debug"`, the mounting of cephfs PVCs will fail. + +Follow the below steps to update the StorageClass's `MountOptions`: + +- Take a backup of the StorageClass using + `kubectl get sc -o yaml > sc.yaml`. +- Edit `sc.yaml` to remove the invalid mount options from `MountOptions` field. +- Delete the StorageClass using `kubectl delete sc `. +- Recreate the StorageClass using `kubectl create -f sc.yaml`. + +Follow the below steps to update the PersistentVolume's `MountOptions`: + +- Identify cephfs PersistentVolumes using + `kubectl get pv | grep `. +- and remove invalid mount options from `MountOptions` field + in the PersistentVolume's using `kubectl edit pv `. + #### Delete removed CephFS PSP, Role and RoleBinding As PSP is deprecated in Kubernetes v1.21.0. Delete PSP related objects as PSP @@ -252,7 +287,7 @@ kubectl delete role cephfs-csi-nodeplugin-psp --ignore-not-found kubectl delete rolebinding cephfs-csi-nodeplugin-psp --ignore-not-found ``` -we have successfully upgraded cephfs csi from v3.7 to v3.8 +we have successfully upgraded cephfs csi from v3.8 to v3.9 ### Upgrading RBD @@ -335,7 +370,7 @@ kubectl delete role rbd-csi-vault-token-review-psp --ignore-not-found kubectl delete rolebinding rbd-csi-vault-token-review-psp --ignore-not-found ``` -we have successfully upgraded RBD csi from v3.7 to v3.8 +we have successfully upgraded RBD csi from v3.8 to v3.9 ### Upgrading NFS @@ -397,7 +432,7 @@ daemonset.apps/csi-nfsplugin configured service/csi-metrics-nfsplugin configured ``` -we have successfully upgraded nfs csi from v3.7 to v3.8 +we have successfully upgraded nfs csi from v3.8 to v3.9 ### CSI Sidecar containers consideration