rbd: add immediate topology flag

[iPraveenParihar]

Describe what this PR does

In csi-external-provisioner version 5.0.1, topology-aware provisioning is enabled by default. As a result, the provisioner now expects topologyKeys to be present in the CSINode object, which must be provided by the user via the --domainlabels flag in the RBD nodeplugin.

Issue: Users upgrading to version 3.12.0 who were not previously using topology-aware provisioning may encounter issues when provisioning RBD PVCs, as the --domainlabels flag might not be set.

Fix: To address this, add --immediate-topology=false to disable topology-aware provisioning. Users requiring topology-aware provisioning should provide the volumeBindingMode as WaitForFirstConsumer and TopologyConstrainedPools in the StorageClass and configure the --domainlabels flag in the RBD nodeplugin.

Fixes: #4777

Future concerns

List items that are not part of the PR and do not impact it's
functionality, but are work items that can be taken up subsequently.

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next major release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

---

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

• /retest ci/centos/<job-name>: retest the <job-name> after unrelated
  failure (please report the failure too!)

[iPraveenParihar]

/test ci/centos/mini-e2e-helm/k8s-1.30

[iPraveenParihar]

@Mergifyio queue

[mergify]

queue

🛑 The pull request has been removed from the queue default

The merge conditions cannot be satisfied due to failing checks.

You can take a look at Queue: Embarked in merge queue check runs for more details.

In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment
@mergifyio refresh on the pull request.

[ceph-csi-bot]

/test ci/centos/upgrade-tests-cephfs

[ceph-csi-bot]

/test ci/centos/upgrade-tests-rbd

[ceph-csi-bot]

/test ci/centos/k8s-e2e-external-storage/1.31

[ceph-csi-bot]

/test ci/centos/mini-e2e-helm/k8s-1.31

[ceph-csi-bot]

/test ci/centos/mini-e2e/k8s-1.31

[ceph-csi-bot]

/test ci/centos/k8s-e2e-external-storage/1.30

[ceph-csi-bot]

/test ci/centos/k8s-e2e-external-storage/1.29

[ceph-csi-bot]

/test ci/centos/mini-e2e-helm/k8s-1.30

[ceph-csi-bot]

/test ci/centos/mini-e2e-helm/k8s-1.29

[ceph-csi-bot]

/test ci/centos/mini-e2e/k8s-1.30

[ceph-csi-bot]

/test ci/centos/mini-e2e/k8s-1.29

[Madhu-1]

/test ci/centos/mini-e2e-helm/k8s-1.30

[Madhu-1]

/test ci/centos/mini-e2e/k8s-1.30

[Madhu-1]

/test ci/centos/upgrade-tests-rbd

[dragoangel]

Looks like --immediate-topology=false not solve things like --feature-gates=Topology=false do, because of errors in csi-provisioner container:

W0822 16:26:29.416084       1 reflector.go:547] k8s.io/client-go/informers/factory.go:160: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:ceph-csi-rbd:ceph-csi-rbd-provisioner" cannot list resource "nodes" in API group "" at the cluster scope
E0822 16:26:29.416289       1 reflector.go:150] k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:ceph-csi-rbd:ceph-csi-rbd-provisioner" cannot list resource "nodes" in API group "" at the cluster scope

It still tries to list node resources, even if they would not be used by csi-provisioner.

If I add to ClusterRole ceph-csi-rbd-provisioner:

+     resources: ["nodes"]
+     verbs: ["get", "list","watch"]
+   - apiGroups: ["storage.k8s.io"]
+     resources: ["csinodes"]
+     verbs: ["get", "list", "watch"]
+   - apiGroups: [""]

which added by default when you set topology.domainLabels - then everything is working. If I would set topology.domainLabels to properly fix rbac - well, nodeplugin will handg with:

missing domain labels [topology.kubernetes.io/region topology.kubernetes.io/zone] on node "node-X"

@Madhu-1 @iPraveenParihar Looks like we need adjust helm chart not always add this permissions for ClusterRole ceph-csi-rbd-provisioner.

[iPraveenParihar]

@dragoangel, Thanks for reporting.

seems like while I was testing helm install script, older clusterroles were not cleaned up properly 😞 and error didn't show up.

[dragoangel]

@dragoangel, Thanks for reporting.

seems like while I was testing helm install script, older clusterroles were not cleaned up properly 😞 and error didn't show up.

Hi @iPraveenParihar assume you had rbac create false, otherwise helm would throw error like you are trying to own resources not owned by this helm chart?