Update Controller To Track Action Progress

[ihcsim]

Change Overview

This PR updates the controller to track the progress of an actionset, by assessing the phase states. The progress tracker runs in its own goroutine with a 2-second ticker. The progress computation assigns weight to a phase, per the underlying Kanister Function the supports the phase. If a Kanister Function is marked as a long-running one, its phase will be assigned the heavy weight. Otherwise, the phase has the normal weight.

The actionset progress always starts at 10%, to distinguish started/running phases from idle ones, and ends at 100%.

The computation is also done on a best-effort basis, with no elaborated retries on update errors, to ensure the progress goroutine doesn't contend for resources nor contribute to performance latency. In other words, it's possible that the progress goes from 10% to 100% with no immediate updates.

Design doc : https://github.com/kanisterio/kanister/blob/master/design/progress-tracking.md

Out-of-scope: Progress computation of each phase based on log outputs are slated for future PRs.

Release note: The actionSet CRD must be upgraded manually, as Helm doesn't upgrade CRD.

Pull request type

Please check the type of change your PR introduces:

• 🚧 Work in Progress
• 🌈 Refactoring (no functional changes, no api changes)
• 🐹 Trivial/Minor
• 🐛 Bugfix
• 🌻 Feature
• 🗺️ Documentation
• 🤖 Test

Issues

• Improve Progress Visibility of Long-Running Tasks #1455

Test Plan

Deploy this multi-actions blueprint, where each phase has different completion time:

cat <<EOF | kubectl apply -f -
apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
  name: actionset-progress
  namespace: kanister
actions:
  action0:
    phases:
    - func: KubeTask
      name: action0-task0
      args:
        namespace: "{{ .Namespace.Name }}"
        image: ghcr.io/kanisterio/kanister-tools:v9.99.9-dev
        command:
        - sh
        - -c
        - |
         sleep 1
    - func: KubeTask
      name: action0-task1
      args:
        namespace: "{{ .Namespace.Name }}"
        image: ghcr.io/kanisterio/kanister-tools:v9.99.9-dev
        command:
        - sh
        - -c
        - |
         sleep 1
    - func: KubeTask
      name: action0-task2
      args:
        namespace: "{{ .Namespace.Name }}"
        image: ghcr.io/kanisterio/kanister-tools:v9.99.9-dev
        command:
        - sh
        - -c
        - |
          sleep 5
    - func: KubeTask
      name: action0-task3
      args:
        namespace: "{{ .Namespace.Name }}"
        image: ghcr.io/kanisterio/kanister-tools:v9.99.9-dev
        command:
        - sh
        - -c
        - |
          sleep 20
  action1:
    phases:
    - func: KubeTask
      name: action1-task0
      args:
        namespace: "{{ .Namespace.Name }}"
        image: ghcr.io/kanisterio/kanister-tools:v9.99.9-dev
        command:
        - sh
        - -c
        - |
          sleep 1
    - func: KubeTask
      name: action1-task1
      args:
        namespace: "{{ .Namespace.Name }}"
        image: ghcr.io/kanisterio/kanister-tools:v9.99.9-dev
        command:
        - sh
        - -c
        - |
          sleep 10
    - func: KubeTask
      name: action1-task2
      args:
        namespace: "{{ .Namespace.Name }}"
        image: ghcr.io/kanisterio/kanister-tools:v9.99.9-dev
        command:
        - sh
        - -c
        - |
          sleep 12
    - func: KubeTask
      name: action1-task3
      args:
        namespace: "{{ .Namespace.Name }}"
        image: ghcr.io/kanisterio/kanister-tools:v9.99.9-dev
        command:
        - sh
        - -c
        - |
          sleep 30
EOF

Create this multi-actions ActionSet to invoke actions in the blueprint:

cat <<EOF | kubectl create -f -
apiVersion: cr.kanister.io/v1alpha1
kind: ActionSet
metadata:
  generateName: multi-actions-
  namespace: kanister
spec:
  actions:
  - name: action0
    blueprint: actionset-progress
    object:
      kind: Namespace
      name: default
  - name: action1
    blueprint: actionset-progress
    object:
      kind: Namespace
      name: default
EOF

The result should look like this:

kubectl -n kanister get actionset multi-actions-t6v65 -ojsonpath='progress: {.status.progress.percentCompleted}{"\n"}last transition time: {.status.progress.lastTransitionTime}
{"\n"}' -w

progrress: 37.50                                                                                                                                                                                 
last transition time: 2022-06-16T16:18:01Z                                                                                                                                                     
progress: 37.50                                                                                                                                                                                 
last transition time: 2022-06-16T16:18:03Z                                                                                                                                                     
progress: 37.50                                                                                                                                                                                 
last transition time: 2022-06-16T16:18:05Z                                                                                                                                                     
progress: 37.50                                                                                                                                                                                 
last transition time: 2022-06-16T16:18:07Z                                                                                                                                                     
progress: 37.50                                                                                                                                                                                 
last transition time: 2022-06-16T16:18:07Z                                                                                                                                                     
progress: 37.50                                                                                                                                                                                 
last transition time: 2022-06-16T16:18:07Z                                                                                                                                                     
progress: 62.50                                                                                                                                                                                 
last transition time: 2022-06-16T16:18:09Z                                                                                                                                                     
percent: 62.50                                            
# ....

• 💪 Manual
• ⚡ Unit test
• 💚 E2E

[github-actions]

Thanks for submitting this pull request 🎉. The team will review it soon and get back to you.

If you haven't already, please take a moment to review our project contributing guideline and Code of Conduct document.

[pavannd1]

1st review with minor nits. Thorough review in progress 🦅 👀

[pavannd1]

@PrasadG193 @viveksinghggits Please review this 👍🏼

[e-sumin]

Correct me if I'm wrong, but seems that err variable is unused ?
in lines #62/#64 we are exiting immediately, and err will stay nil

[ihcsim]

Good 👁️. Yes, it can be removed.

[viveksinghggits]

Should we add progress in the additional printer columns so that someone can just figure out the progress by looking at the output of k get actionset.

[ihcsim]

@viveksinghggits that's a good idea.

Now we can watch the progress while the actionset is running:

$ watch k -n kanister get actionset                                      
NAME                  PROGRESS   LAST TRANSITION TIME   STATE
multi-actions-zmfj4   100.00     2022-07-20T23:41:11Z   complete
multi-actions-fl5w9   87.50      2022-07-21T02:33:12Z   running

[pavannd1]

LGTM 🚀