Refactor controller with pkg runtime helpers #342

phillebaba · 2021-10-19T08:48:35Z

Fixes #322

HelmRelease should implement the interfaces from conditions, and use this package to work with conditions.
HelmReleaseReconciler should implement the helpers from controller, and use these to record Events and Metrics.
HelmReleaseReconciler should exit early without recording any status when .spec.suspend is defined.
The patch package should be implemented to (safely) patch resources without running into conflicts.
The kstatus Reconciling and Stalled conditions should be properly implemented to provide better signals.

phillebaba · 2021-10-27T12:29:57Z

api/v2beta1/helmrelease_types.go

-	// +kubebuilder:validation:Enum=ChartVersion;Revision
-	// +kubebuilder:default:=ChartVersion
-	// +optional
-	ReconcileStrategy string `json:"reconcileStrategy,omitempty"`


This was removed from the HelmChart so I assume that it should be removed from here also.

My guess is that the answer is yes.

Missed this comment for some reason :-S

No, it was introduced to the HelmChart resource in a later version than the reconcilers-dev branch. So you don't have access to the API against this version, but it will return.

Ok good, will keep this open and loop back when it becomes relevant.

phillebaba · 2021-10-27T12:31:15Z

controllers/helmrelease_controller.go

+			),
+			conditions.WithNegativePolarityConditions(
+				v2.RemediatedCondition,
+			),


All of this is probably wrong, need to go back and read up on conditions.

This should contain a list of all conditions that have an influence on the end result of Ready. What defines if something is (or isn't) a negative polarity is if the condition highlights normal vs abnormal behavior, a condition type can't be both negative and positive (and should thus only be listed once).

Examples:

Ready == True is "normal behavior", as the condition is present and True if everything is operating as expected.

Stalled == True is "abnormal behavior" as the condition is present and True if an abnormality occurs, but removed from the resource once resolved.

RemediatedCondition is a negative polarity condition, as it being present indicates an error and should result in an inverse of the True and return a Ready == False.

So for now it looks right, but I will have to come back in a later PR and make changes to the different conditions.

controllers/helmrelease_controller.go

phillebaba · 2021-10-27T18:03:27Z

@hiddeco I am getting a hang of most things. Things seem to work in a full deployment now. The most confusing thing right now is the different conditions and the preferred methods. For example the differences between using released conditon and ready condition.

Need to work some of these things to make sure that we are not breaking existing functionality.

phillebaba · 2021-11-02T15:27:58Z

controllers/helmrelease_controller_chart.go

-			ValuesFile:        template.Spec.ValuesFile,
+			Interval:    template.GetInterval(hr.Spec.Interval),
+			ValuesFiles: template.Spec.ValuesFiles,
+			ValuesFile:  template.Spec.ValuesFile,


Should Suspend and AccessFrom be added to here?

Introduction of suspend would be an idea, but also raises a question: if a user decides to add both fields, the one of the chart won't be reconciled. Is this expected behavior, or an awkward result of design decisions?

For AccessFrom, @stefanprodan will likely be able to tell more.

I think the suspend feature should be in its own PR to keep things simple. But yes the question about setting both fields is an important one. Is it due to legacy that we have both field options or is there some other reason behind this? My suggestion for now is to just throw an error if both are set.

Given that the controller generates the HelmChart, AccessFrom should be set to allow the namespace where the HelmRelease is. Sadly it's impossible to to restrict the access to a single namespace on Kubernetes < 1.21 since the kubernetes.io/metadata.name label was aded in 1.21. Given this I would set this to allow all.

phillebaba · 2021-11-02T15:30:24Z

controllers/helmrelease_controller.go

-		r.recordReadiness(ctx, hr)
-	}
+	// TODO: Progressing is set immediately on reconcile so is it necessary to set it here?
+	hr, _ = v2.HelmReleaseAttempted(hr, revision, releaseRevision, valuesChecksum)


Should hasNewState be used here?

To keep most of the logic (that should be rewritten as well as noted in (an) other issue(s)) intact I would maintain it for now.

Would the easiest solution be to skip the status patching and only implement the failure counter reset?

controllers/helmrelease_controller.go

api/v2beta1/helmrelease_types.go

controllers/helmrelease_controller.go

hiddeco · 2021-11-08T11:54:36Z

controllers/helmrelease_controller_chart.go

-			ValuesFile:        template.Spec.ValuesFile,
+			Interval:    template.GetInterval(hr.Spec.Interval),
+			ValuesFiles: template.Spec.ValuesFiles,
+			ValuesFile:  template.Spec.ValuesFile,


Introduction of suspend would be an idea, but also raises a question: if a user decides to add both fields, the one of the chart won't be reconciled. Is this expected behavior, or an awkward result of design decisions?

For AccessFrom, @stefanprodan will likely be able to tell more.

hiddeco · 2021-11-08T11:59:07Z

controllers/helmrelease_controller.go

+			),
+			conditions.WithNegativePolarityConditions(
+				v2.RemediatedCondition,
+			),


This should contain a list of all conditions that have an influence on the end result of Ready. What defines if something is (or isn't) a negative polarity is if the condition highlights normal vs abnormal behavior, a condition type can't be both negative and positive (and should thus only be listed once).

Examples:

Ready == True is "normal behavior", as the condition is present and True if everything is operating as expected.

Stalled == True is "abnormal behavior" as the condition is present and True if an abnormality occurs, but removed from the resource once resolved.

RemediatedCondition is a negative polarity condition, as it being present indicates an error and should result in an inverse of the True and return a Ready == False.

controllers/helmrelease_controller.go

hiddeco · 2021-11-08T12:07:28Z

controllers/helmrelease_controller.go

-		r.recordReadiness(ctx, hr)
-	}
+	// TODO: Progressing is set immediately on reconcile so is it necessary to set it here?
+	hr, _ = v2.HelmReleaseAttempted(hr, revision, releaseRevision, valuesChecksum)


To keep most of the logic (that should be rewritten as well as noted in (an) other issue(s)) intact I would maintain it for now.

controllers/helmrelease_controller.go

Signed-off-by: Philip Laine <philip.laine@xenit.se>

darkowlzz · 2021-12-02T13:56:46Z

controllers/helmrelease_controller.go

-	if reconcileErr != nil {
-		r.event(ctx, hr, hc.GetArtifact().Revision, events.EventSeverityError,
-			fmt.Sprintf("reconciliation failed: %s", reconcileErr.Error()))
+	if result, err := r.reconcileRelease(ctx, obj, chart, values); reconcileErr != nil {


I've been investigating the reason for the test failure in CI and found that because of reconcileErr here, the error from reconcileRelease() is always ignored. So the install-fail test doesn't gets expected result.

Suggested change

if result, err := r.reconcileRelease(ctx, obj, chart, values); reconcileErr != nil {

if result, err := r.reconcileRelease(ctx, obj, chart, values); err != nil {

This may not resolve the CI failures, there's more to it. Investigating...

This should fix most of the CI failures, but I found some more conditionals related issues. Will comment them inline.

darkowlzz

I tried reproducing the e2e tests that are failing in the CI and left comments about what I did to make it work. Also left a few comments around the status conditions usage. Hope those comments will make it clear some of the conditionals usage patterns.

darkowlzz · 2021-12-02T22:29:56Z

controllers/helmrelease_controller.go

-		r.event(ctx, hr, hr.Status.LastAttemptedRevision, events.EventSeverityError, msg)
-		return v2.HelmReleaseNotReady(hr, v2.ArtifactFailedReason, msg), ctrl.Result{Requeue: true}, reconcileErr
+		obj.IncrementFailureCounter()
+		conditions.MarkFalse(obj, meta.ReadyCondition, v2.ArtifactFailedReason, "Chart reconcilliation failed: %s", reconcileErr.Error())


Since ready condition is calculated as a summary based on other conditions, it'd be better to mark ReleasedCondition false here. Setting ready condition here gets overwritten in the Reconcile() when ready condition is set as a summary of other conditions.

darkowlzz · 2021-12-02T22:31:08Z

controllers/helmrelease_controller.go

 		msg := fmt.Sprintf("HelmChart '%s/%s' is not ready", hc.GetNamespace(), hc.GetName())
-		r.event(ctx, hr, hr.Status.LastAttemptedRevision, events.EventSeverityInfo, msg)
+		obj.IncrementFailureCounter()
+		conditions.MarkFalse(obj, meta.ReadyCondition, v2.ArtifactFailedReason, msg)


Same as above, this can also be ReleasedCondition marked as false.

darkowlzz · 2021-12-02T22:32:25Z

controllers/helmrelease_controller.go

+		if err := r.checkDependencies(ctx, obj); err != nil {
+			msg := fmt.Sprintf("dependencies do not meet ready condition (%s), retrying in %s", err.Error(), r.requeueDependency.String())
+			obj.IncrementFailureCounter()
+			conditions.MarkFalse(obj, meta.ReadyCondition, v2.GetHelmChartFailedReason, msg)


Same as above, this can also be ReleasedCondition marked as false.

darkowlzz · 2021-12-02T22:32:54Z

controllers/helmrelease_controller.go

-		r.event(ctx, hr, hr.Status.LastAttemptedRevision, events.EventSeverityError, err.Error())
-		return v2.HelmReleaseNotReady(hr, v2.InitFailedReason, err.Error()), ctrl.Result{Requeue: true}, nil
+		obj.IncrementFailureCounter()
+		conditions.MarkFalse(obj, meta.ReadyCondition, v2.InitFailedReason, "could not get chart values: %s", err.Error())


Same as above, this can also be ReleasedCondition marked as false.

darkowlzz · 2021-12-02T22:33:52Z

controllers/helmrelease_controller.go

-		r.event(ctx, hr, hr.Status.LastAttemptedRevision, events.EventSeverityError, err.Error())
-		return v2.HelmReleaseNotReady(hr, v2.ArtifactFailedReason, err.Error()), ctrl.Result{Requeue: true}, nil
+		obj.IncrementFailureCounter()
+		conditions.MarkFalse(obj, meta.ReadyCondition, v2.ArtifactFailedReason, "could not get chart artifact: %s", err.Error())


Same as above, this can also be ReleasedCondition marked as false.

darkowlzz · 2021-12-02T22:34:10Z

controllers/helmrelease_controller.go

-			fmt.Sprintf("reconciliation failed: %s", reconcileErr.Error()))
+	if result, err := r.reconcileRelease(ctx, obj, chart, values); reconcileErr != nil {
+		obj.IncrementFailureCounter()
+		conditions.MarkFalse(obj, meta.ReadyCondition, v2.ArtifactFailedReason, "could not get chart artifact: %s", err.Error())


Same as above, this can also be ReleasedCondition marked as false.

darkowlzz · 2021-12-02T22:38:51Z

controllers/helmrelease_controller.go

 			err = fmt.Errorf("install retries exhausted")
-			return v2.HelmReleaseNotReady(hr, released.Reason, err.Error()), err
+			conditions.MarkFalse(hr, meta.ReadyCondition, released.Reason, err.Error())


This can be RemediatedCondition.

Signed-off-by: Philip Laine <philip.laine@xenit.se>

phillebaba marked this pull request as draft October 19, 2021 08:48

phillebaba force-pushed the feature/pkg-runtime-helpers branch from c6f257c to 7383ae1 Compare October 25, 2021 08:31

stefanprodan changed the title ~~Reactor helm controller with pkg runtime helpers~~ Refactor controller with pkg runtime helpers Oct 25, 2021

phillebaba force-pushed the feature/pkg-runtime-helpers branch 3 times, most recently from 2b1a3df to e265e9d Compare October 27, 2021 09:20

phillebaba commented Oct 27, 2021

View reviewed changes

controllers/helmrelease_controller.go Outdated Show resolved Hide resolved

phillebaba changed the base branch from main to reconcilers-dev November 2, 2021 15:19

phillebaba marked this pull request as ready for review November 2, 2021 15:19

phillebaba force-pushed the feature/pkg-runtime-helpers branch 3 times, most recently from c581997 to 06415a3 Compare November 2, 2021 15:24

phillebaba commented Nov 2, 2021

View reviewed changes

phillebaba force-pushed the feature/pkg-runtime-helpers branch 2 times, most recently from 7b175ee to 451c234 Compare November 2, 2021 15:36

phillebaba commented Nov 2, 2021

View reviewed changes

api/v2beta1/helmrelease_types.go Show resolved Hide resolved

darkowlzz reviewed Nov 4, 2021

View reviewed changes

api/v2beta1/helmrelease_types.go Show resolved Hide resolved

phillebaba force-pushed the feature/pkg-runtime-helpers branch from 451c234 to 4b42969 Compare November 5, 2021 16:46

hiddeco self-assigned this Nov 5, 2021

hiddeco reviewed Nov 8, 2021

View reviewed changes

phillebaba force-pushed the feature/pkg-runtime-helpers branch from 660342a to a3130b5 Compare November 15, 2021 13:08

Refactor controller with pkg runtime helpers

ccba028

Signed-off-by: Philip Laine <philip.laine@xenit.se>

phillebaba force-pushed the feature/pkg-runtime-helpers branch from a3130b5 to ccba028 Compare December 1, 2021 13:12

Remove test success from ready condition

2776b25

Signed-off-by: Philip Laine <philip.laine@xenit.se>

darkowlzz reviewed Dec 2, 2021

View reviewed changes

phillebaba force-pushed the feature/pkg-runtime-helpers branch from 8153265 to 19a7285 Compare December 10, 2021 13:46

phillebaba force-pushed the feature/pkg-runtime-helpers branch 4 times, most recently from d20927e to dfd9c2e Compare December 10, 2021 15:13

phillebaba force-pushed the feature/pkg-runtime-helpers branch 3 times, most recently from 5984c7c to 6e33965 Compare December 21, 2021 15:50

phillebaba force-pushed the feature/pkg-runtime-helpers branch from 6e33965 to d096400 Compare January 10, 2022 14:52

Fix conditions and error check

6389ad7

Signed-off-by: Philip Laine <philip.laine@xenit.se>

phillebaba force-pushed the feature/pkg-runtime-helpers branch from d096400 to 6389ad7 Compare January 13, 2022 14:10

phillebaba merged commit f286591 into reconcilers-dev Jan 13, 2022

phillebaba deleted the feature/pkg-runtime-helpers branch January 13, 2022 14:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor controller with pkg runtime helpers #342

Refactor controller with pkg runtime helpers #342

phillebaba commented Oct 19, 2021 •

edited

Loading

phillebaba Oct 27, 2021

phillebaba Nov 9, 2021

hiddeco Nov 9, 2021

phillebaba Nov 9, 2021

phillebaba Oct 27, 2021

hiddeco Nov 8, 2021

phillebaba Nov 16, 2021

phillebaba commented Oct 27, 2021

phillebaba Nov 2, 2021

hiddeco Nov 8, 2021

phillebaba Nov 9, 2021

stefanprodan Nov 9, 2021

phillebaba Nov 2, 2021

hiddeco Nov 8, 2021

phillebaba Nov 9, 2021

hiddeco Nov 8, 2021

hiddeco Nov 8, 2021

hiddeco Nov 8, 2021

darkowlzz Dec 2, 2021 •

edited

Loading

darkowlzz Dec 2, 2021 •

edited

Loading

darkowlzz Dec 2, 2021

darkowlzz left a comment

darkowlzz Dec 2, 2021

darkowlzz Dec 2, 2021

darkowlzz Dec 2, 2021

darkowlzz Dec 2, 2021

darkowlzz Dec 2, 2021

darkowlzz Dec 2, 2021

darkowlzz Dec 2, 2021

	if result, err := r.reconcileRelease(ctx, obj, chart, values); reconcileErr != nil {
	if result, err := r.reconcileRelease(ctx, obj, chart, values); err != nil {

Refactor controller with pkg runtime helpers #342

Refactor controller with pkg runtime helpers #342

Conversation

phillebaba commented Oct 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phillebaba commented Oct 27, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darkowlzz Dec 2, 2021 • edited Loading

Choose a reason for hiding this comment

darkowlzz Dec 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darkowlzz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phillebaba commented Oct 19, 2021 •

edited

Loading

darkowlzz Dec 2, 2021 •

edited

Loading

darkowlzz Dec 2, 2021 •

edited

Loading