Progress tracking actionset #1586

viveksinghggits · 2022-08-05T14:29:24Z

Change Overview

Re-opens #1487, with #1576 fix.

Pull request type

Please check the type of change your PR introduces:

Issues

fixes [BUG] Kanister Controller cannot update Status on Kubernetes versions < v1.20 #1576

Test Plan

💪 Manual
⚡ Unit test

/kanister/pkg/progress (progress-tracking-actionset*) » go test                       
OK: 6 passed
PASS
ok  	github.com/kanisterio/kanister/pkg/progress	0.026s

Install Kanister on k8s version 1.19 and 1.21 to check if we are backup and restore an application.

kctx: kind-kind
~/work/opensource/kanister/pkg/progress (progress-tracking-actionset*) » k version 
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-21T23:01:33Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

kctx: kind-kind
~/work/opensource/kanister/pkg/progress (progress-tracking-actionset*) » k get actionsets.cr.kanister.io -n kanister
NAME                         PROGRESS   LAST TRANSITION TIME   STATE
backup-22dtd                 100.00     2022-08-05T14:16:49Z   complete
backup-bl5w2                 100.00     2022-08-05T14:16:33Z   complete
restore-backup-22dtd-stq9j   100.00     2022-08-05T14:31:19Z   complete

kctx: kind-kind
~/work/opensource/kanister/examples/mysql (progress-tracking-actionset*) » k version 
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2022-05-19T20:23:31Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.22) and server (1.19) exceeds the supported minor version skew of +/-1

kctx: kind-kind
~/work/opensource/kanister/examples/mysql (progress-tracking-actionset*) » k get actionsets.cr.kanister.io -n kanister 
NAME                         PROGRESS   LAST TRANSITION TIME   STATE
backup-jpldg                 100.00     2022-08-05T14:39:43Z   complete
restore-backup-jpldg-vjlvp   100.00     2022-08-05T14:40:04Z   complete

Signed-off-by: Ivan Sim <ivan.sim@kasten.io>

In the k8s cluster < 1.20 we got into a problem while updating actionset progress. Details can be found [here](#1576). This tries to fix that.

github-actions · 2022-08-05T14:29:46Z

Thanks for submitting this pull request 🎉. The team will review it soon and get back to you.

If you haven't already, please take a moment to review our project contributing guideline and Code of Conduct document.

pkg/progress/action_single_phase_test.go

pkg/apis/cr/v1alpha1/types.go

PrasadG193 · 2022-08-08T04:29:08Z

pkg/progress/action.go

+	"github.com/kanisterio/kanister/pkg/log"
+	"github.com/kanisterio/kanister/pkg/reconcile"
+	"github.com/kanisterio/kanister/pkg/validate"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"


Third-party pkg imports should be in separate group

PrasadG193 · 2022-08-08T04:33:27Z

pkg/progress/action.go

+	return updateActionSet(ctx, client, actionSet, progressPercent, now)
+}
+
+func weight(phase *crv1alpha1.BlueprintPhase) float64 {


[nit] ideally function name should start with a verb. e.g calculateWeight or getWeight

PrasadG193 · 2022-08-08T04:35:39Z

pkg/progress/action.go

+	return weightNormal
+}
+
+func longRunning(phase *crv1alpha1.BlueprintPhase) bool {


[nit] since the function returns bool - we can rename it to something - containsLongRunningFunc

PrasadG193 · 2022-08-08T04:36:55Z

pkg/progress/action.go

+	return nil
+}
+
+func completedOrFailed(actionSet *crv1alpha1.ActionSet) bool {


[nit] Since the func returns bool, we can rename this to something like - isCompletedOrFailed

PrasadG193 · 2022-08-08T04:40:46Z

pkg/progress/action.go

+					"nextUpdateTime": time.Now().Add(pollDuration),
+				}
+				log.Error().WithError(err).Print("failed to update phase progress", fields)
+				continue


This may run into an infinite loop if the client fails to update action progress every iteration. Instead of continue, should we return error?

The idea here is to ignore intermediate, flakey errors, due to e.g., API server or client-go throttling. The final progress should always be calculated based on the actionset status. If the actionset status completed, progress should always be 100% despite of intermediate errors.

It shouldn't lead to an infinite loop because either ctx will be done, or the completedOrFailed() function will return, based on the actionset status.

PrasadG193 · 2022-08-08T04:50:54Z

pkg/progress/action.go

+		"namespace": actionSet.GetNamespace(),
+		"progress":  progressPercent,
+	}
+	log.Debug().Print("updating action progress", fields)


We should also add actionset name (and may be phase name as well) to the logs so that we can differentiate between logs in case of multiple statements

PrasadG193 · 2022-08-08T04:51:43Z

pkg/progress/action.go

+	if err := reconcile.ActionSet(ctx, client.CrV1alpha1(), actionSet.GetNamespace(), actionSet.GetName(), updateFunc); err != nil {
+		return err
+	}
+
+	return nil


Suggested change

if err := reconcile.ActionSet(ctx, client.CrV1alpha1(), actionSet.GetNamespace(), actionSet.GetName(), updateFunc); err != nil {

return err

}

return nil

return reconcile.ActionSet(ctx, client.CrV1alpha1(), actionSet.GetNamespace(), actionSet.GetName(), updateFunc)

error check can be skipped

PrasadG193 · 2022-08-08T04:55:10Z

pkg/apis/cr/v1alpha1/types.go

@@ -131,6 +132,16 @@ type ActionStatus struct {
 	DeferPhase Phase `json:"deferPhase,omitempty"`
 }

+// ActionProgress provides information on the progress of an action.
+type ActionProgress struct {


should we also add the name of the current phase in progress?

phase progress is planned for the next iteration. at this level, we just want to communicate the overall progress without bloating the API spec.

- Improve test a bit - Add `omitempty` to another field of `actionProgress`

pavannd1

I believe most of the changes in this PR are a part of the revert and were previously approved. My approval of this PR is purely focused on the intended bug fix. We can follow up on the other review comments in a separate PR.

ihcsim

@viveksinghggits Thanks for working on this 👍. Since the previous PR has been reviewed quite a bit, we should keep changes to this PR to the minimal, unless more bugs are found. If possible, I want to avoid another round of review churns - that last PR took a month (or more?) to merge.

I suggest other important non-bug review items be added to a new GH issue, which we may or may not address when we start the phase progress work.

We should start finding ways to surface these kind of errors sooner in the dev cycle. Reverting PRs is costly and inefficient as they create churns and re-works.

* Update controller to track weighted action progress Signed-off-by: Ivan Sim <ivan.sim@kasten.io> * Revert auto-changes by codegen Signed-off-by: Ivan Sim <ivan.sim@kasten.io> * Address Eugen's feedback Signed-off-by: Ivan Sim <ivan.sim@kasten.io> * Address Pavan's feedback Signed-off-by: Ivan Sim <ivan.sim@kasten.io> * Address Vivek's feedback Signed-off-by: Ivan Sim <ivan.sim@kasten.io> * Fix tests Signed-off-by: Ivan Sim <ivan.sim@kasten.io> * Fix a bug while updating lastTransitionTime in <1.20 cluster In the k8s cluster < 1.20 we got into a problem while updating actionset progress. Details can be found [here](kanisterio#1576). This tries to fix that. * Remove extra newline from codegen.sh * Address review comments - Improve test a bit - Add `omitempty` to another field of `actionProgress` Co-authored-by: Ivan Sim <ivan.sim@kasten.io> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

ihcsim and others added 9 commits June 16, 2022 07:40

Update controller to track weighted action progress

d7ac0b2

Signed-off-by: Ivan Sim <ivan.sim@kasten.io>

Revert auto-changes by codegen

942ee9f

Signed-off-by: Ivan Sim <ivan.sim@kasten.io>

Address Eugen's feedback

0d0d941

Signed-off-by: Ivan Sim <ivan.sim@kasten.io>

Address Pavan's feedback

afc2e76

Signed-off-by: Ivan Sim <ivan.sim@kasten.io>

Address Vivek's feedback

9662130

Signed-off-by: Ivan Sim <ivan.sim@kasten.io>

Fix tests

316c88b

Signed-off-by: Ivan Sim <ivan.sim@kasten.io>

Merge branch 'master' into progress-tracking-actionset

a30649e

Fix a bug while updating lastTransitionTime in <1.20 cluster

1e782ca

In the k8s cluster < 1.20 we got into a problem while updating actionset progress. Details can be found [here](#1576). This tries to fix that.

Remove extra newline from codegen.sh

16b0e6b

infraq added this to In Progress in Kanister Aug 5, 2022

viveksinghggits requested review from PrasadG193, ihcsim and pavannd1 August 5, 2022 14:29

pavannd1 reviewed Aug 5, 2022

View reviewed changes

pkg/progress/action_single_phase_test.go Show resolved Hide resolved

pkg/apis/cr/v1alpha1/types.go Show resolved Hide resolved

PrasadG193 reviewed Aug 8, 2022

View reviewed changes

Address review comments

2bd05f3

- Improve test a bit - Add `omitempty` to another field of `actionProgress`

Kanister automation moved this from In Progress to Reviewer approved Aug 8, 2022

pavannd1 approved these changes Aug 8, 2022

View reviewed changes

ihcsim approved these changes Aug 8, 2022

View reviewed changes

viveksinghggits added the kueue label Aug 8, 2022

Merge branch 'master' into progress-tracking-actionset

46a0917

mergify bot merged commit 34c76cd into master Aug 8, 2022

mergify bot deleted the progress-tracking-actionset branch August 8, 2022 20:26

Kanister automation moved this from Reviewer approved to Done Aug 8, 2022

ihcsim mentioned this pull request Aug 9, 2022

Improve Progress Visibility of Long-Running Tasks #1455

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Progress tracking actionset #1586

Progress tracking actionset #1586

viveksinghggits commented Aug 5, 2022 •

edited

Loading

github-actions bot commented Aug 5, 2022

PrasadG193 Aug 8, 2022

PrasadG193 Aug 8, 2022

PrasadG193 Aug 8, 2022

PrasadG193 Aug 8, 2022

PrasadG193 Aug 8, 2022

ihcsim Aug 8, 2022

ihcsim Aug 8, 2022

PrasadG193 Aug 8, 2022

PrasadG193 Aug 8, 2022

PrasadG193 Aug 8, 2022

ihcsim Aug 8, 2022

pavannd1 left a comment •

edited

Loading

ihcsim left a comment

Progress tracking actionset #1586

Progress tracking actionset #1586

Conversation

viveksinghggits commented Aug 5, 2022 • edited Loading

Change Overview

Pull request type

Issues

Test Plan

github-actions bot commented Aug 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavannd1 left a comment • edited Loading

Choose a reason for hiding this comment

ihcsim left a comment

Choose a reason for hiding this comment

viveksinghggits commented Aug 5, 2022 •

edited

Loading

pavannd1 left a comment •

edited

Loading