RBD e2e test failure

[Madhu-1]

delayed, please verify that the provisioner is running and correctly registered.
  I0912 11:09:26.152543 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:26.152558 81108 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (pvc-90df8359-1b81-48f5-8e20-00f8a2b5bf99) volume UUID (1a473b5d-03e8-486a-8cdf-b7025e12ebde) volume omap name ()
  I0912 11:09:28.148770 81108 pvc.go:63] waiting for PVC rbd-pvc (576 seconds elapsed)
  I0912 11:09:28.152917 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:28.152957 81108 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0912 11:09:28.152968 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:28.152978 81108 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (pvc-90df8359-1b81-48f5-8e20-00f8a2b5bf99) volume UUID (1a473b5d-03e8-486a-8cdf-b7025e12ebde) volume omap name ()
  I0912 11:09:30.148573 81108 pvc.go:63] waiting for PVC rbd-pvc (578 seconds elapsed)
  I0912 11:09:30.152945 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:30.152982 81108 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0912 11:09:30.152995 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:30.153008 81108 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (pvc-90df8359-1b81-48f5-8e20-00f8a2b5bf99) volume UUID (1a473b5d-03e8-486a-8cdf-b7025e12ebde) volume omap name ()
  I0912 11:09:32.148237 81108 pvc.go:63] waiting for PVC rbd-pvc (580 seconds elapsed)
  I0912 11:09:32.152433 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:32.152476 81108 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0912 11:09:32.152491 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:32.152495 81108 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (pvc-90df8359-1b81-48f5-8e20-00f8a2b5bf99) volume UUID (1a473b5d-03e8-486a-8cdf-b7025e12ebde) volume omap name ()
  I0912 11:09:34.148419 81108 pvc.go:63] waiting for PVC rbd-pvc (582 seconds elapsed)
  I0912 11:09:34.155659 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:34.155700 81108 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0912 11:09:34.155709 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:34.155734 81108 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (pvc-90df8359-1b81-48f5-8e20-00f8a2b5bf99) volume UUID (1a473b5d-03e8-486a-8cdf-b7025e12ebde) volume omap name ()
  I0912 11:09:36.148710 81108 pvc.go:63] waiting for PVC rbd-pvc (584 seconds elapsed)
  I0912 11:09:36.152884 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:36.152922 81108 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0912 11:09:36.152934 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:36.152947 81108 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (pvc-90df8359-1b81-48f5-8e20-00f8a2b5bf99) volume UUID (1a473b5d-03e8-486a-8cdf-b7025e12ebde) volume omap name ()
  I0912 11:09:38.148750 81108 pvc.go:63] waiting for PVC rbd-pvc (586 seconds elapsed)
  I0912 11:09:38.152977 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:38.153021 81108 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0912 11:09:38.153031 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:38.153045 81108 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (pvc-90df8359-1b81-48f5-8e20-00f8a2b5bf99) volume UUID (1a473b5d-03e8-486a-8cdf-b7025e12ebde) volume omap name ()
  I0912 11:09:40.148550 81108 pvc.go:63] waiting for PVC rbd-pvc (588 seconds elapsed)
  I0912 11:09:40.152937 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:40.152971 81108 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0912 11:09:40.152982 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:40.152997 81108 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (pvc-90df8359-1b81-48f5-8e20-00f8a2b5bf99) volume UUID (1a473b5d-03e8-486a-8cdf-b7025e12ebde) volume omap name ()
  I0912 11:09:42.148498 81108 pvc.go:63] waiting for PVC rbd-pvc (590 seconds elapsed)
  I0912 11:09:42.152953 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:42.152992 81108 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0912 11:09:42.153007 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:42.153019 81108 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (pvc-90df8359-1b81-48f5-8e20-00f8a2b5bf99) volume UUID (1a473b5d-03e8-486a-8cdf-b7025e12ebde) volume omap name ()
  I0912 11:09:44.148657 81108 pvc.go:63] waiting for PVC rbd-pvc (592 seconds elapsed)
  I0912 11:09:44.153191 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:44.153226 81108 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0912 11:09:44.153242 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:44.153253 81108 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (pvc-90df8359-1b81-48f5-8e20-00f8a2b5bf99) volume UUID (1a473b5d-03e8-486a-8cdf-b7025e12ebde) volume omap name ()
  I0912 11:09:46.147948 81108 pvc.go:63] waiting for PVC rbd-pvc (594 seconds elapsed)
  I0912 11:09:46.152437 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:46.152481 81108 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0912 11:09:46.152495 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:46.152513 81108 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (pvc-90df8359-1b81-48f5-8e20-00f8a2b5bf99) volume UUID (1a473b5d-03e8-486a-8cdf-b7025e12ebde) volume omap name ()
  I0912 11:09:48.147939 81108 pvc.go:63] waiting for PVC rbd-pvc (596 seconds elapsed)
  I0912 11:09:48.152191 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:48.152226 81108 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0912 11:09:48.152240 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:48.152252 81108 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (pvc-90df8359-1b81-48f5-8e20-00f8a2b5bf99) volume UUID (1a473b5d-03e8-486a-8cdf-b7025e12ebde) volume omap name ()
  I0912 11:09:50.148614 81108 pvc.go:63] waiting for PVC rbd-pvc (598 seconds elapsed)
  I0912 11:09:50.152834 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:50.152875 81108 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0912 11:09:50.152890 81108 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-401/rbd-pvc"
  I0912 11:09:50.152905 81108 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (pvc-90df8359-1b81-48f5-8e20-00f8a2b5bf99) volume UUID (1a473b5d-03e8-486a-8cdf-b7025e12ebde) volume omap name ()

There seems to be a corner case happening in RBD E2E , Logs are available here https://jenkins-ceph-csi.apps.ocp.cloud.ci.centos.org/blue/rest/organizations/jenkins/pipelines/mini-e2e_k8s-1.31/runs/63/nodes/94/steps/97/log/?start=0 for debugging