ceph-csi 3.12.0 not able to provision RBD PVC.

[Madhu-1]

          > @iPraveenParihar can you please check above?

#4776 should fix the CSI NodePlugin with v.3.12.0 driver fails when missing domain labels deployed via helm.

this does indeed "fixes" the mentioned problem, however, on new pvc, there's new one :D

We yet to decide on how to enable/disable topology aware provisioning (#4726).
For now, the workaround is to add --feature-gates=Topology=false for csi-provisioner in csi-rbdplugin-provisioner deployment pod.

@ypyly, Can you please try with above workaround and see if that helps you?

Originally posted by @iPraveenParihar in #4775 (comment)

[ypyly]

as commented in the original issue adding upper-mentioned arg to the provisioner does solve the issue.
P.S> for the helm installation, you need to add

provisioner:
  provisioner:
    extraArgs:
      - 'feature-gates=Topology=false'

to your values

[iPraveenParihar]

well its actually for both helm/manual deployment.

let me summarise this -
v3.12.0 Ceph-CSI uses csi-external-provisioner: v5.0.1 which has topology aware provisioning enabled by default.
And the csi-external-provisioner will need topologyKeys (domain labels passed to csi-rbdplugin in csi-rbdplugin daemonset pod) to generate AccessibilityRequirements and pass it to CreateVolume request.

I made changes in #4726 to add an option enable/disable topology from Ceph-CSI. But it isn't good as users would want some StorageClasses to use topology based provisioning, others not which is handled by the same Ceph-CSI driver.

other solution is that we could have --immediate-topology=false set in csi-external-provisioner.

1. If topology based provisioning is not needed - set volumeBindingMode as Immediate in StorageClass
2. If topology based provisioning is needed - set volumeBindingMode as WaitForFirstConsumer in StorageClass and pass --domainlabels

@Madhu-1 @nixpanic, WDYT of above?

[Madhu-1]

well its actually for both helm/manual deployment.

let me summarise this - v3.12.0 Ceph-CSI uses csi-external-provisioner: v5.0.1 which has topology aware provisioning enabled by default. And the csi-external-provisioner will need topologyKeys (domain labels passed to csi-rbdplugin in csi-rbdplugin daemonset pod) to generate AccessibilityRequirements and pass it to CreateVolume request.

I made changes in #4726 to add an option enable/disable topology from Ceph-CSI. But it isn't good as users would want some StorageClasses to use topology based provisioning, others not which is handled by the same Ceph-CSI driver.

other solution is that we could have --immediate-topology=false set in csi-external-provisioner.

1. If topology based provisioning is not needed - set volumeBindingMode as Immediate in StorageClass
2. If topology based provisioning is needed - set volumeBindingMode as WaitForFirstConsumer in StorageClass and pass --domainlabels

@Madhu-1 @nixpanic, WDYT of above?

IMO above 1, 2 is the right options when to use and when not to use topology based provisioning. we need to avoid introducing new flags to enable and disable as in same cluster we need to support both, i am fine to make domainLabel as mandatory if required. but we also need to consider how it will impact non-k8s as well.