validate drpolicy s3 profiles and scheduling interval #310

hatfieldbrian · 2021-10-09T17:44:25Z

This pull request:

moves scheduling interval format validation from drpc and vrg reconcilers to drpolicy and vrg creation
adds s3 profile connectivity validation to drpolicy reconciler using existing s3 store connection routine and a new status condition
adds unit tests for the previous two items
exports ObjectStoreGetter and ObjectStorer interfaces and their methods so they can be implemented in a different package, e.g. controllers_test, but leaves the s3 implementations' methods not directly callable outside of controllers as their types,
s3ObjectStoreGetter and s3ObjectStore, are not exported.

Close #304

Signed-off-by: hatfieldbrian <bhatfiel@redhat.com>

BenamarMk · 2021-10-11T14:02:31Z

controllers/drpolicy_controller.go

+
+	if condition := util.DrpolicyValidatedConditionGet(drpolicy); condition != nil {
+		if condition.Status == metav1.ConditionTrue {
+			log.Info(`valid -> valid`)


Curious; why do you use backticks instead of quotes? Backticks are for handling special characters. https://golang.org/ref/spec#String_literals
To keep the code common, I suggest using "" unless necessary.

I just discovered backticks. I chose them since these strings do not require interpretation. One might say that double quotes are for handling special characters, e.g. newlines, tabs, etc.

@hatfieldbrian are you planning to change the backticks to double quotes or not? The more I look at it, the more I want it to be changed given that they are not providing any special addition.

I'm willing to adopt this style for the future. Since this pull request has already been merged I opened #317 to track changing the code merged in this pull request.

raghavendrabhat · 2021-10-11T16:11:51Z

controllers/volumereplicationgroup_controller.go

-		// No requeue, as there is no reconcile till user changes desired spec to a valid value
-		return ctrl.Result{}, nil
-	}
-


Probably we should not remove this from VRG?

I agree with not having multiple different implementations for validating schedule in different controllers and instead better to use a single implementation probably done in utils. However, I think that it is still better to have a validation check for schedule in VRG. It is because, VRG spec has a field for schedulingInterval. Hence in this case VRG would be doing validation of its own spec field. For DRPC it would be correct to remove the validation of schedule from there as it was validating the schedule from DRPolicy spec and DRPC spec in itself does not have the schedulingInterval field.

Let me know your thoughts on this.

vrg.spec.schedulingInterval format should still be validated. The change to vrg's types.go file should cause it to be validated in the create/update path. The same change was made to drpolicy's types.go file. A test case was added for drpolicy but not duplicated for vrg.

ShyamsundarR

Approving, as we can fix the comments subsequently (minor changes). The backtick that @BenamarMk points out is confusing as it is rarely used. (again if not changed now, can be a subsequent patch @BenamarMk ?)

ShyamsundarR · 2021-10-19T12:03:46Z

controllers/drplacementcontrol_controller.go

-	err = r.validateSchedule(drPolicy)
-	if err != nil {
-		return ctrl.Result{}, err
+	if err := rmnutil.DrpolicyValidated(drPolicy); err != nil {


Note for later: We need to update DRPC status with some information from this point for user consumption of errors.

ShyamsundarR · 2021-10-19T12:15:09Z

controllers/drpolicy_controller.go

+	)
+
+	if condition := util.DrpolicyValidatedConditionGet(drpolicy); condition != nil {
+		if condition.Status == metav1.ConditionTrue {


(future) If ramen config changed (which is not detected or reconciled, hence assume it was done async). We will not rerun the validation. Should we consider running this every time, or at least when the generation of the condition and the resource are different?

I agree it should be possible to transition from valid to invalid if say an s3 profile stored in ramen config, or the secret that is points to changes. It seems the right way to do that is by drpolicy watching for changes to a cluster's ramen config and s3 secrets referenced by an s3 profile referenced in the drpolicy.

Or it could become invalid if a drpolicy changed to point to a different s3 profile, which would result in a drpolicy reconcile call, but not a revalidation. Though, I understand drpolicy modification is not supported at this point and it or parts of it may be made immutable. For example, removing clusters could result in managed cluster vrg clusterroles being leaked.

However, say s3 access permission is revoked for the user specified in an s3 profile's secret. This seems to be a way for the s3 store connectivity to become invalid without a change to any kubernetes resources and hence no normal way for drpolicy reconciler to be run.

But I do think the point is to validate a drpolicy's s3 profiles and so previously valid ones should be re-validated on any changes to them that drpolicy can be made aware of by normal means.

A workaround to revalidate is to delete and recreate a drpolicy, or if it is mutable, delete its validated condition or modify its validated condition's state to false or unknown which are both considered "invalid".

#319 tracks this issue

ShyamsundarR · 2021-10-19T12:15:59Z

controllers/drpolicy_controller.go

+			return client.Status().Update(ctx, drpolicy)
+		}
+		conditionSetFalse = func(reason string, err error) error {
+			log.Info(`invalid -> invalid`)


If a prior S3 profile was incorrect and was subsequently corrected, the error message won't change for the next incorrect profile?

NOTE: If we decide to update the condition, we should take care not to update the same messages and just change the timestamp, as this has caused frequent reconciles with VRG in the past.

You're right. It should change. Also, good point about updating conditionally when the message changes. Issue #318 tracks this.

BenamarMk

@hatfieldbrian we'll merge this as is. But please change the backticks later

Syncing latest changes from upstream main for ramen

validate drpolicy s3 profiles and scheduling interval

b77ba6c

Signed-off-by: hatfieldbrian <bhatfiel@redhat.com>

BenamarMk reviewed Oct 11, 2021

View reviewed changes

raghavendrabhat reviewed Oct 11, 2021

View reviewed changes

ShyamsundarR approved these changes Oct 19, 2021

View reviewed changes

BenamarMk approved these changes Oct 19, 2021

View reviewed changes

BenamarMk merged commit 1d3cba5 into RamenDR:main Oct 19, 2021

This was referenced Oct 20, 2021

double quote each string literal unless its backslash escapes should not be interpreted #317

Open

update drpolicy validated status condition's message if it changes #318

Closed

hatfieldbrian deleted the 304 branch October 20, 2021 19:43

hatfieldbrian mentioned this pull request Oct 20, 2021

drpolicy re-reconcile on modify of s3 profiles and secrets #319

Closed

hatfieldbrian mentioned this pull request Nov 23, 2021

DRPolicy.Spec.SchedulingInterval validation move from DRPC reconciler to DRPolicy reconciler #201

Closed

raghavendra-talur pushed a commit to raghavendra-talur/ramen that referenced this pull request Jul 30, 2024

Merge pull request RamenDR#310 from red-hat-storage/sync_us--main

870ff96

Syncing latest changes from upstream main for ramen

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

validate drpolicy s3 profiles and scheduling interval #310

validate drpolicy s3 profiles and scheduling interval #310

hatfieldbrian commented Oct 9, 2021

BenamarMk Oct 11, 2021

hatfieldbrian Oct 11, 2021

BenamarMk Oct 19, 2021 •

edited

Loading

hatfieldbrian Oct 20, 2021 •

edited

Loading

raghavendrabhat Oct 11, 2021

hatfieldbrian Oct 11, 2021

ShyamsundarR left a comment

ShyamsundarR Oct 19, 2021

ShyamsundarR Oct 19, 2021

hatfieldbrian Oct 20, 2021 •

edited

Loading

hatfieldbrian Oct 20, 2021

ShyamsundarR Oct 19, 2021

hatfieldbrian Oct 20, 2021

BenamarMk left a comment

validate drpolicy s3 profiles and scheduling interval #310

validate drpolicy s3 profiles and scheduling interval #310

Conversation

hatfieldbrian commented Oct 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenamarMk Oct 19, 2021 • edited Loading

Choose a reason for hiding this comment

hatfieldbrian Oct 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShyamsundarR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hatfieldbrian Oct 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenamarMk left a comment

Choose a reason for hiding this comment

BenamarMk Oct 19, 2021 •

edited

Loading

hatfieldbrian Oct 20, 2021 •

edited

Loading

hatfieldbrian Oct 20, 2021 •

edited

Loading