filter with default VRC annotation if multiple VRCs detected #1476

Rakshith-R · 2024-06-27T10:55:32Z

This commit modifies selectVolumeReplicationClass() to
filter VRCs list with default VRC annotation
replication.storage.openshift.io/volume-replication-class: true
if multiple VRCs are matched.
This makes sure we select the default VRC when multiple VRCs of
same schedule and provisioner but a special default annotated VRC
is present in the cluster.

ShyamsundarR

Leaving some initial comments, even though it is marked as a draft.

ShyamsundarR · 2024-06-30T11:10:40Z

controllers/vrg_volrep.go

+			matchingReplicationClassList = append(matchingReplicationClassList, replicationClass)
+		}
+	}
+


We can deal with 0 or 1 found VRClasses here and call filterDefaultVRC if not.

ShyamsundarR · 2024-06-30T11:12:34Z

controllers/vrg_volrep.go

+			provisioner, v.instance.Spec.Async.SchedulingInterval))
+
+		return nil, fmt.Errorf("no VolumeReplicationClass found to match provisioner and schedule")
+	case 1:


len could also be > 1 if there are multiple classes marked as the default.

If we have multiple vrcs with default annotation, we should fail in line 1324.

controllers/vrg_volrep.go

test/addons/rbd-mirror/start

ShyamsundarR · 2024-06-30T11:16:52Z

test/addons/rbd-mirror/vrc-flatten-sample.yaml

+apiVersion: replication.storage.openshift.io/v1alpha1
+kind: VolumeReplicationClass
+metadata:
+  name: vrc-sample


Name is duplicate of existing VRClass and hence will overwrite the default (based on order of apply called).

This needs to be addressed, else when used it will cause unwanted troubleshooting cycles to figure out the names are in conflict.

nirs

Thanks @Rakshith-R!

nirs · 2024-06-30T16:35:21Z

test/addons/rbd-mirror/vrc-sample.yaml

@@ -6,6 +6,8 @@ apiVersion: replication.storage.openshift.io/v1alpha1
 kind: VolumeReplicationClass
 metadata:
  name: vrc-sample
+  annotation:
+    replication.storage.openshift.io/is-default-class: true


true is yaml true, but annotations are map[string]string. Even if this works, better use "true" to make it more clear.

@Rakshith-R can we use the string "true" here as stated?

nirs · 2024-06-30T16:37:10Z

test/addons/rbd-mirror/start

@@ -268,4 +277,7 @@ wait_until_pool_mirroring_is_healthy(cluster2)
 deploy_vrc_sample(cluster1)
 deploy_vrc_sample(cluster2)

+deploy_vrc_flatten_sample(cluster1)
+deploy_vrc_flatten_sample(cluster2)


Would be nice to group all the resources we deploy in a kustomzation and apply all of them in one command. Then we don't need to helpers to apply the yamsl.

test/addons/rbd-mirror/start

nirs · 2024-06-30T16:42:46Z

controllers/vrg_volrep.go

+			provisioner, v.instance.Spec.Async.SchedulingInterval))
+
+		return nil, fmt.Errorf("no VolumeReplicationClass found to match provisioner and schedule")
+	case 1:


If we have multiple vrcs with default annotation, we should fail in line 1324.

nirs · 2024-06-30T16:45:16Z

controllers/vrg_volrep.go

@@ -28,6 +28,11 @@ import (
 	rmnutil "github.com/ramendr/ramen/controllers/util"
 )

+const (
+	// defaultVRCAnnotationKey is the default annotation key for VolumeReplicationClass
+	defaultVRCAnnotationKey = "replication.storage.opensds.io/volume-replication-class"


Based on changes to vrc-sample.yaml, this should be:

"replication.storage.openshift.io/is-default-class"

Should we use a label instead of annotation? This make is possible to list the default vrcs - we don't need this now, but we may want to do this in the future.

nirs · 2024-06-30T16:49:09Z

controllers/vrg_volrep.go


-		return nil, fmt.Errorf("failed to get the storageclass of pvc %s (%w)",
-			namespacedName, err)
+		return nil, fmt.Errorf("failed to get the storageclass of pvc %s (%w)", namespacedName, err)


This seems like unrelated formatting changes, it would be better not to modify anything which is not needed by the actual change.

nirs · 2024-06-30T16:50:21Z

controllers/vrg_volrep.go

+}
+
+// filterDefaultVRC filters the VRC list to return VRCs with default annotation
+// if the list contains moret then one VRC.


typo: moret

nirs · 2024-06-30T16:52:07Z

controllers/vrg_volrep.go

@@ -1271,17 +1276,52 @@ func (v *VRGInstance) selectVolumeReplicationClass(

 		// ReplicationClass that matches both VRG schedule and pvc provisioner
 		if schedulingInterval == v.instance.Spec.Async.SchedulingInterval {
-			v.log.Info(fmt.Sprintf("Found VolumeReplicationClass that matches provisioner and schedule %s/%s",
-				storageClass.Provisioner, v.instance.Spec.Async.SchedulingInterval))


Removing the log makes sense, but we can do this in another commit. Lets focus on adding filtering multiple vrcs using the default annotation in this change.

nirs · 2024-06-30T17:09:10Z

controllers/vrg_volrep.go

 		}
+	} else {
+		defaultReplicationClassList = replicationClassList
+	}


Based on the name, I expect filterDefaultVRC to accept list with more than one vrc, and return a list with only the matching vrcs based the default annotation. The caller can do the handling (0, 1, >1). This will keep this function simpler, and we will cal it only when we find multiple matching vrcs.

The flow can be:

matchingVRCs := v.selectVolumeReplicationClass() if len(matchingVRCs) > 1 { matchingVRCs = filterDefaultVRC(matchingVRCs) }

Now we can succeed if we have have list of one vrc, or fail in all other cases.

nirs · 2024-06-30T17:09:49Z

controllers/vrg_volrep.go

 		}
+	} else {
+		defaultReplicationClassList = replicationClassList


This is not the default, so maybe rename the list to filteredVRCs?

Rakshith-R · 2024-07-03T10:20:59Z

(related) We can add the check for v.vrcUpdated inside the function updateReplicationClassList and fix the invocation from this function as well updatePVCList to reduce cognitive complexity of this function.

This just helped me a lot just slipping past the function line count limit.

The pr is now ready for review.
Added only env tests for now.

nirs

I only looked at the filtering part, but see also comments from previous version that are not addressed by the current version.

nirs · 2024-07-03T10:51:56Z

internal/controller/vrg_volrep.go

 		}
 	}

-	v.log.Info(fmt.Sprintf("No VolumeReplicationClass found to match provisioner and schedule %s/%s",
+	filteredVRCs := v.filterDefaultVRC(matchingReplicationClassList)


If we have more than one vrc and none have the default annotation, this call will return empty list and we will fail with the "No vrc found" instead of "multiple vrcs found".

It will help if filterVRCs() will be smarter about selecting the default annotation, and return the right error when having no default annotation or multiple vrcs with default annotation.

Also it would be easier to follow if we first handle the zero case and then handle the > 1 case, and finish with the expected case. It is easier to handle errors correctly when handling errors first.

Something like:

if len == 0: fail if len > 1: select the default or fail return found vrc

Yes, that's by design.
Filtering with annotation will be triggered when there's more one matching VRC.
We are logging a info msg about filtering in the call. It's implied that more than one VRC was found but none had default annotation. Ramen anyway falls back to volsync when it doesn't find a VRC so It does not exactly bubble out the error persistently to anyone..

Hence, we can again encounter case of 0 VRCs being found.

That's why the 0 check is at the end.

The code here when used does not default to volsync and updates a status condition with the error. Hence being specific here helps the user understand the issue than looking at logs.

For example the createVR path would generate an event if an error is returned from here.

nirs · 2024-07-03T11:16:06Z

internal/controller/vrg_volrep.go

+) []*volrep.VolumeReplicationClass {
+	if len(replicationClassList) <= 1 {
+		return replicationClassList
+	}


It will be clean to call this only when there are more than 1, so this check is not needed.

It will be clean to call this only when there are more than 1, so this check is not needed.

I think this logic can indeed be part of the filterVRC() function itself.
The selectVRC() cannot get any bigger.

nirs · 2024-07-03T11:18:23Z

internal/controller/vrg_volrep.go

+		}
+	}
+
+	return filteredVRCs


Here we can report the right error:

multiple vrcs, none marked as default

multiple vrcs, more than one marked as default

This will help to simplify the caller, and very simple and clear here - the only function dealing with multiple vrcs.

answered in another comment above.

#1476 (comment)

Rakshith-R · 2024-07-03T11:40:34Z

I only looked at the filtering part, but see also comments from previous version that are not addressed by the current version.

#1476 (comment)
#1476 (comment)

I think these should answer most of the comments which were given towards the prev(draft) version.
Changes in [test/addons/rbd-mirror/start file are no longer part of this pr.

Rakshith-R · 2024-07-09T15:03:31Z

/cc @ShyamsundarR

ShyamsundarR

Has this been tested in a drenv setup, with an additional VRClass with the default annotation set?

Suggested patch to address the need for filterDefaultVRC to only deal with filtering when there are multiple VRCs detected:

diff --git a/internal/controller/vrg_volrep.go b/internal/controller/vrg_volrep.go
index 9df9b737..0528bb07 100644
--- a/internal/controller/vrg_volrep.go
+++ b/internal/controller/vrg_volrep.go
@@ -1275,36 +1275,27 @@ func (v *VRGInstance) selectVolumeReplicationClass(
                }
        }
 
-       filteredVRCs := v.filterDefaultVRC(matchingReplicationClassList)
-
-       switch len(filteredVRCs) {
+       switch len(matchingReplicationClassList) {
        case 0:
-               v.log.Info(fmt.Sprintf("No VolumeReplicationClass found to match provisioner and schedule %s/%s",
+               v.log.Info(fmt.Sprintf("No VolumeReplicationClass found that match provisioner and schedule %s/%s",
                        storageClass.Provisioner, v.instance.Spec.Async.SchedulingInterval))
 
-               return nil, fmt.Errorf("no VolumeReplicationClass found to match provisioner and schedule")
+               return nil, fmt.Errorf("no VolumeReplicationClass found that match provisioner and schedule")
        case 1:
                v.log.Info(fmt.Sprintf("Found VolumeReplicationClass that matches provisioner and schedule %s/%s",
                        storageClass.Provisioner, v.instance.Spec.Async.SchedulingInterval))
 
-               return filteredVRCs[0], nil
+               return matchingReplicationClassList[0], nil
        }
 
-       v.log.Info(fmt.Sprintf("Found multiple VolumeReplicationClass that matches provisioner and schedule %s/%s",
-               storageClass.Provisioner, v.instance.Spec.Async.SchedulingInterval))
-
-       return nil, fmt.Errorf("multiple VolumeReplicationClass found to match provisioner and schedule")
+       return v.filterDefaultVRC(matchingReplicationClassList)
 }
 
-// filterDefaultVRC filters the VRC list to return VRCs with default annotation
+// filterDefaultVRC filters the VRC list to return VRC with default annotation
 // if the list contains more than one VRC.
 func (v *VRGInstance) filterDefaultVRC(
        replicationClassList []*volrep.VolumeReplicationClass,
-) []*volrep.VolumeReplicationClass {
-       if len(replicationClassList) <= 1 {
-               return replicationClassList
-       }
-
+) (*volrep.VolumeReplicationClass, error) {
        v.log.Info("Found multiple matching VolumeReplicationClasses, filtering with default annotation")
 
        filteredVRCs := []*volrep.VolumeReplicationClass{}
@@ -1317,7 +1308,19 @@ func (v *VRGInstance) filterDefaultVRC(
                }
        }
 
-       return filteredVRCs
+       switch len(filteredVRCs) {
+       case 0:
+               v.log.Info(fmt.Sprintf("Multiple VolumeReplicationClass found, with no default annotation (%s/%s)",
+                       replicationClassList[0].Spec.Provisioner, v.instance.Spec.Async.SchedulingInterval))
+
+               return nil, fmt.Errorf("multiple VolumeReplicationClass found, with no default annotation, %s",
+                       defaultVRCAnnotationKey)
+       case 1:
+               return filteredVRCs[0], nil
+       }
+
+       return nil, fmt.Errorf("multiple VolumeReplicationClass found with default annotation, %s",
+               defaultVRCAnnotationKey)
 }
 
 // getStorageClass inspects the PVCs being protected by this VRG instance for the passed in namespacedName, and

ShyamsundarR · 2024-07-10T12:53:29Z

internal/controller/vrg_volrep.go

@@ -28,6 +28,11 @@ import (
 	rmnutil "github.com/ramendr/ramen/internal/controller/util"
 )

+const (
+	// defaultVRCAnnotationKey is the default annotation key for VolumeReplicationClass
+	defaultVRCAnnotationKey = "replication.storage.openshift.io/volume-replication-class"


(future) It would be useful if this was imported fro csi-addons, such that it does not have to be redefined here.

ShyamsundarR · 2024-07-10T12:54:58Z

test/addons/rbd-mirror/vrc-flatten-sample.yaml

+apiVersion: replication.storage.openshift.io/v1alpha1
+kind: VolumeReplicationClass
+metadata:
+  name: vrc-sample


This needs to be addressed, else when used it will cause unwanted troubleshooting cycles to figure out the names are in conflict.

ShyamsundarR · 2024-07-10T12:55:34Z

test/addons/rbd-mirror/vrc-sample.yaml

@@ -6,6 +6,8 @@ apiVersion: replication.storage.openshift.io/v1alpha1
 kind: VolumeReplicationClass
 metadata:
  name: vrc-sample
+  annotation:
+    replication.storage.openshift.io/is-default-class: true


@Rakshith-R can we use the string "true" here as stated?

ShyamsundarR · 2024-07-10T13:12:57Z

internal/controller/vrg_volrep.go

 		}
 	}

-	v.log.Info(fmt.Sprintf("No VolumeReplicationClass found to match provisioner and schedule %s/%s",
+	filteredVRCs := v.filterDefaultVRC(matchingReplicationClassList)


The code here when used does not default to volsync and updates a status condition with the error. Hence being specific here helps the user understand the issue than looking at logs.

For example the createVR path would generate an event if an error is returned from here.

This commit modifies selectVolumeReplicationClass() to filter VRCs list with default VRC annotation `replication.storage.opensds.io/volume-replication-class: true` if multiple VRCs are matched. This makes sure we select the default VRC when multiple VRCs of same schedule and provisioner but a special default annotated VRC is present in the cluster. Signed-off-by: Rakshith R <rar@redhat.com>

Rakshith-R · 2024-07-11T07:28:03Z

I only looked at the filtering part, but see also comments from previous version that are not addressed by the current version.

#1476 (comment) #1476 (comment)

I think these should answer most of the comments which were given towards the prev(draft) version. Changes in [test/addons/rbd-mirror/start file are no longer part of this pr.

Sorry, i had forgotten to remove the files from test/addons/rbd-mirror/start

I tested it on ODF cluster with changes ramen image and multiple VRCs.
It picks up the annotated one as expected and fails in other negative cases.

2024-07-11T06:40:55.246Z	INFO	controllers.VolumeReplicationGroup.vrginstance	controller/vrg_volrep.go:1299	Found multiple matching VolumeReplicationClasses, filtering with default annotation	{"VolumeReplicationGroup": {"name":"app-busybox-rbd-3-push-placement-drpc","namespace":"app-busybox-rbd-3-push"}, "rid": "18739824-3b09-42d5-a915-797e183e0fdb", "State": "primary"}
2024-07-11T06:40:55.246Z	INFO	controllers.VolumeReplicationGroup.vrginstance	controller/vrg_volrep.go:1313	Multiple VolumeReplicationClass found, with no default annotation (openshift-storage.rbd.csi.ceph.com/5m)	{"VolumeReplicationGroup": {"name":"app-busybox-rbd-3-push-placement-drpc","namespace":"app-busybox-rbd-3-push"}, "rid": "18739824-3b09-42d5-a915-797e183e0fdb", "State": "primary"}
...

nirs

There are some detail we can improve, but looks good enough for now.

Rakshith-R force-pushed the filterDefaultVRC branch 4 times, most recently from b8201ef to ce21847 Compare June 28, 2024 10:51

ShyamsundarR reviewed Jun 30, 2024

View reviewed changes

nirs reviewed Jun 30, 2024

View reviewed changes

nirs requested a review from raghavendra-talur July 1, 2024 14:41

Rakshith-R force-pushed the filterDefaultVRC branch 5 times, most recently from d2ea8ea to 1a888da Compare July 3, 2024 10:05

Rakshith-R marked this pull request as ready for review July 3, 2024 10:21

Rakshith-R requested review from nirs and ShyamsundarR July 3, 2024 10:21

nirs reviewed Jul 3, 2024

View reviewed changes

Rakshith-R requested a review from nirs July 3, 2024 11:40

ShyamsundarR reviewed Jul 10, 2024

View reviewed changes

Rakshith-R force-pushed the filterDefaultVRC branch from 1a888da to f4b360f Compare July 11, 2024 07:12

Rakshith-R force-pushed the filterDefaultVRC branch from f4b360f to c75bf84 Compare July 11, 2024 07:21

Rakshith-R closed this Jul 11, 2024

Rakshith-R reopened this Jul 11, 2024

Rakshith-R requested a review from ShyamsundarR July 11, 2024 07:29

nirs approved these changes Jul 17, 2024

View reviewed changes

ShyamsundarR approved these changes Jul 17, 2024

View reviewed changes

ShyamsundarR merged commit 97f1457 into RamenDR:main Jul 17, 2024
31 checks passed

filter with default VRC annotation if multiple VRCs detected #1476

filter with default VRC annotation if multiple VRCs detected #1476

Conversation

Rakshith-R commented Jun 27, 2024 • edited Loading

ShyamsundarR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nirs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rakshith-R commented Jul 3, 2024

nirs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rakshith-R commented Jul 3, 2024

Rakshith-R commented Jul 9, 2024

ShyamsundarR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rakshith-R commented Jul 11, 2024 • edited Loading

nirs left a comment

Choose a reason for hiding this comment

Rakshith-R commented Jun 27, 2024 •

edited

Loading

Rakshith-R commented Jul 11, 2024 •

edited

Loading