-
Notifications
You must be signed in to change notification settings - Fork 152
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
RFC: Add design for blueprint and phase progress calculation (#1461)
* Add design RFC for blueprint and phase progress calculation Signed-off-by: Ivan Sim <ivan.sim@kasten.io> * Address Dave's and Vivek's feedback Signed-off-by: Ivan Sim <ivan.sim@kasten.io> * Address feedback from sync-up meeting Signed-off-by: Ivan Sim <ivan.sim@kasten.io> * Add script to generate table of content Signed-off-by: Ivan Sim <ivan.sim@kasten.io> * Improve wording Signed-off-by: Ivan Sim <ivan.sim@kasten.io> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
- Loading branch information
1 parent
4678669
commit 3ec3967
Showing
2 changed files
with
314 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,263 @@ | ||
# Blueprint And Phase Progress Tracking | ||
|
||
<!-- toc --> | ||
- [Problem Statement](#problem-statement) | ||
- [Proposed Solution](#proposed-solution) | ||
- [Summary](#summary) | ||
- [Assumptions And Constraints](#assumptions-and-constraints) | ||
- [Changes To API](#changes-to-api) | ||
- [Progress Tracking](#progress-tracking) | ||
- [Action Progress](#action-progress) | ||
- [Phase Progress](#phase-progress) | ||
- [Error Handling](#error-handling) | ||
- [Test Cases](#test-cases) | ||
<!-- /toc --> | ||
|
||
## Problem Statement | ||
|
||
When an actionset triggers a long-running task like the `CopyVolumeData` | ||
function, the only way to determine any sort of progress is to gain direct access | ||
to the logs and events of the task pod. This may not always be feasible as the | ||
user may not have the appropriate RBAC permissions to access these subresource | ||
endpoints in the pod's namespace. If the export operation takes a long time, | ||
it's also possible that the pod might be terminated prematurely without leaving | ||
behind any traces of logs to indicate how far along the task was. | ||
|
||
Being able to persist progress data will be very helpful for both live reporting | ||
of task progress, as well as future retrospection (e.g., the latency of every | ||
phase of an action). | ||
|
||
## Proposed Solution | ||
|
||
### Summary | ||
|
||
The `ActionSet` CRD's `status` subresource will be updated with new fields to | ||
communicate the progress of the action and its phases to the user. The | ||
`kube.ExecOutput()` and `kube.Task()` interfaces will be updated to accept a new | ||
progress I/O writer. | ||
|
||
### Assumptions And Constraints | ||
|
||
* The progress computation should not compromise the main data protection task's | ||
latency nor lead to resource contention. Progress computation will be performed | ||
on a best-effort basis, where it may be de-prioritized with no guarantee on the | ||
accuracy of its result, or skipped entirely in the event of resource contention. | ||
|
||
### Changes To API | ||
|
||
An action overall progress will be reported under the new `status.progress` | ||
field of the `ActionSet` resource. The progress of each phase will be included | ||
in the phase's subsection as `status.actions[*].phases[*].progress`. | ||
|
||
For example, | ||
|
||
```yaml | ||
status: | ||
progress: | ||
percentCompleted: 50.00 # 1 out of 2 actions are completed | ||
lastTransitionTime: 2022-04-06 14:23:34 | ||
actions: | ||
- blueprint: my-blueprint | ||
name: action-00 | ||
phases: | ||
- name: echo | ||
state: completed | ||
progress: | ||
percentCompleted: 100.00 | ||
lastTransitionTime: 2022-04-06 14:13:00 | ||
extraStats: | ||
transferredBytes: 20KiB | ||
processedBytes: 15KiB | ||
readBytes: 120KiB | ||
totalBytes: 120KiB | ||
- name: echo | ||
state: completed | ||
progress: | ||
percentCompleted: 100.00 | ||
lastTransitionTime: 2022-04-06 14:23:34 | ||
name: action-01 | ||
phases: | ||
- name: echo | ||
state: pending | ||
progress: | ||
percentCompleted: 30.00 | ||
lastTransitionTime: 2022-04-06 14:30:31 | ||
state: pending | ||
``` | ||
|
||
### Progress Tracking | ||
|
||
Since progress tracking may not be meaningful for short-lived tasks, we will | ||
limit the initial implementation effort to the following Kanister Functions | ||
which normally are used to invoke long-running operations: | ||
|
||
* `BackupData` | ||
* `BackupDataAll` | ||
* `RestoreData` | ||
* `RestoreDataAll` | ||
* `CopyVolumeData` | ||
* `CreateRDSSnapshot` | ||
* `ExportRDSSnapshotToLocation` | ||
* `RestoreRDSSnapshot` | ||
|
||
#### Action Progress | ||
|
||
Initially, the progress of an action is computed by checking the number of | ||
completed phases against the total number of phases within the action: | ||
|
||
``` | ||
action_progress_percent = num_completed_phases / total_num_phases * 100 | ||
``` | ||
|
||
In subsequent implementation, the computation alogrithm can be updated to assign | ||
more weight to phases with long-running operations. It's also possible to post | ||
periodic progress updates using an exponential backoff mechanism as long as the | ||
underlying phases are still alive. | ||
|
||
When an action starts, its progress will be updated to 10%, instead of keeping | ||
it at 0%. This will help to distinguish between in-progress action from an | ||
inactive one. | ||
|
||
The action's progress will only be set to 100% after all its phases completed | ||
without failures. The action's progress should never exceed 100%. | ||
|
||
#### Phase Progress | ||
|
||
As each phase within a blueprint may involve executing different commands | ||
producing different outputs, this design proposes a phase progress tracking | ||
interface that can use different "trackers" to map command outputs to numeric | ||
progress status. | ||
|
||
Some example trackers include ones that can track progress by: | ||
|
||
* checking the number of uploaded bytes against estimated total bytes | ||
* checking the duration elapsed against the estimated duration to complete the | ||
operation | ||
* parsing the log outputs for milestone events to indicate the 25%, 50%, 75% and | ||
100% markers | ||
|
||
✍️ Currently, Kanister Functions do not utilize Kopia to perform their | ||
underlying work. Once the work to integrate Kopia into Kanister is completed, | ||
we can extract the progress status directly from the log outputs. | ||
|
||
Here's a sample log output of the Kopia create snapshot function: | ||
|
||
```sh | ||
$ kopia snapshot create kanister | ||
Snapshotting isim@pop-os:/home/isim/workspace/kanisterio/kanister ... | ||
- 5 hashing, 4186 hashed (329.1 MB), 0 cached (0 B), uploaded 309.8 MB, estimated 2 GB (16.3%) 3m38s left | ||
``` | ||
|
||
Kanister Functions that are currently using Restic already have a set of library | ||
functions that can be used to extract progress status from the Restic logs. See | ||
e.g., the `restic.SnapshotStatsFromBackupLog()` function in | ||
[`pkg/restic/restic.go`](https://github.com/kanisterio/kanister/blob/c5acaac88a60c22faeadd59c49b20942f662331d/pkg/restic/restic.go#L362) | ||
|
||
Since all the long-running functions rely on the `KubeExec` and `KubeTask` | ||
functions, most implementation changes will be done on these two functions. | ||
|
||
✍️ Defer phase should also included in the phase-level progress tracking. | ||
|
||
Here's an example code snippet of the proposed interface written in Go: | ||
|
||
```go | ||
// ./pkg/progress/phase | ||
package phase | ||
|
||
type ProgressTracker struct { | ||
t Tracker | ||
R Result | ||
} | ||
|
||
type Result struct { | ||
StatusInPercent chan string | ||
Err chan error | ||
} | ||
|
||
func (pt *ProgressTracker) Write(p []byte) (n int, err error) { | ||
if err := pt.t.Compute(string(p), pt); err != nil { | ||
return len(p), err | ||
} | ||
|
||
return len(p), nil | ||
} | ||
|
||
func (pt *ProgressTracker) Result() <-chan string { | ||
return pt.R.StatusInPercent | ||
} | ||
|
||
func (pt *ProgressTracker) Err() <-chan error { | ||
return pt.R.Err | ||
} | ||
|
||
type Tracker interface { | ||
Compute(cmdOutput string, p *ProgressTracker) error | ||
} | ||
``` | ||
|
||
This is an example of what the client code would look like: | ||
|
||
```go | ||
ctx, cancel := context.WithCancel(context.Background()) | ||
defer cancel() | ||
|
||
bytesTracker := BytesTracker { totalNumBytes: 268435456 } | ||
progressTracker := phase.NewProgressTracker(bytesTracker) | ||
go func() { | ||
for { | ||
select { | ||
case <-context.Done(): | ||
// handle context.Err() | ||
return | ||
case err := <-progressTracker.Err(): | ||
// handle err | ||
return | ||
case r := <-progressTracker.Result(): | ||
// update the actionset's status with r. | ||
// might need some more refactoring in order to return | ||
// this to ./pkg/controller/controller.go | ||
} | ||
} | ||
}() | ||
|
||
out := io.MultiWriter(os.Stdout, progressTracker) | ||
kube.ExecOutput(cli, namespace, pod, container, command, in, out, errw) | ||
``` | ||
|
||
Here's an sample tracker that computes progress status based on the amount of | ||
data uploaded vs. the total amount of data: | ||
|
||
```go | ||
var _ Tracker = (*BytesTracker)(nil) | ||
|
||
type BytesTracker struct { | ||
totalNumBytes int64 | ||
} | ||
|
||
func (b BytesTracker) Compute(cmdOutput string, t *ProgressTracker) error { | ||
totalNumBytesUploaded, err := parse(cmdOutput) | ||
if err != nil { | ||
return err | ||
} | ||
|
||
pt.R.StatusInPercent = totalNumBytesUploaded/totalNumBytes * 100 | ||
return nil | ||
} | ||
``` | ||
|
||
#### Error Handling | ||
|
||
If a phase failed, the progress tracking will cease immediately. The last | ||
reported progress will be retained in the actionset's `status` subresource. | ||
|
||
## Test Cases | ||
|
||
New unit tests to be added to the new `progress` package to cover blueprint | ||
progress, and phase progress calculated with a sample tracker: | ||
|
||
* Blueprint with single phase: | ||
* Completed successfully - assert that blueprint and phase progress are at 100% | ||
* Failed to finish - assert that blueprint progress and phase progress are at 10% | ||
* Blueprint with multiple phases: | ||
* Completed successfully - assert that blueprint and phase progress are at 100% | ||
* Failed to finish at different phases - assert that progress calculation is correct |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,51 @@ | ||
#!/usr/bin/env bash | ||
|
||
# Copyright 2019 The Kubernetes Authors. | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
set -o errexit | ||
set -o nounset | ||
set -o pipefail | ||
|
||
# keep in sync with hack/verify-toc.sh | ||
TOOL_VERSION=b8c54a57d69f29386d055584e595f38d65ce2a1f | ||
|
||
# cd to the root path | ||
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd -P)" | ||
cd "${ROOT}" | ||
|
||
# create a temporary directory | ||
TMP_DIR=$(mktemp -d) | ||
|
||
# cleanup | ||
exitHandler() ( | ||
echo "Cleaning up..." | ||
rm -rf "${TMP_DIR}" | ||
) | ||
trap exitHandler EXIT | ||
|
||
# Perform go install in a temp dir as we are not tracking this version in a go | ||
# module. | ||
# If we do the go install in the repo, it will create/update go.mod and go.sum. | ||
cd "${TMP_DIR}" | ||
GO111MODULE=on GOBIN="${TMP_DIR}" go install "sigs.k8s.io/mdtoc@${TOOL_VERSION}" | ||
export PATH="${TMP_DIR}:${PATH}" | ||
cd "${ROOT}" | ||
|
||
# Update tables of contents if necessary. | ||
find ./design -name '*.md' \ | ||
| xargs mdtoc --inplace --max-depth=5 || ( | ||
echo "Failed generating TOC. If this failed silently and you are on mac, try 'brew install grep'" | ||
exit 1 | ||
) |