Skip to content

Commit

Permalink
Update task for DaemonSet history and rollback (#4098)
Browse files Browse the repository at this point in the history
* Update task for DaemonSet history and rollback

Also remove mentions of templateGeneration field because it's deprecated

* Address comments
  • Loading branch information
janetkuo authored and Jessica Yao committed Sep 22, 2017
1 parent 3a26ab5 commit 4871d62
Show file tree
Hide file tree
Showing 3 changed files with 168 additions and 39 deletions.
1 change: 1 addition & 0 deletions _data/tasks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,7 @@ toc:
- title: Manage Cluster Daemons
section:
- docs/tasks/manage-daemon/update-daemon-set.md
- docs/tasks/manage-daemon/rollback-daemon-set.md

- title: Manage GPUs
section:
Expand Down
154 changes: 154 additions & 0 deletions docs/tasks/manage-daemon/rollback-daemon-set.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
---
assignees:
- janetkuo
title: Performing a Rollback on a DaemonSet
---

{% capture overview %}

This page shows how to perform a rollback on a DaemonSet.

{% endcapture %}


{% capture prerequisites %}

* The DaemonSet rollout history and DaemonSet rollback features are only
supported in `kubectl` in Kubernetes version 1.7 or later.
* Make sure you know how to [perform a rolling update on a
DaemonSet](/docs/tasks/manage-daemon/update-daemon-set/)

{% endcapture %}


{% capture steps %}

## Performing a Rollback on a DaemonSet

### Step 1: Find the DaemonSet revision you want to roll back to

You can skip this step if you just want to roll back to the last revision.

List all revisions of a DaemonSet:

```shell
kubectl rollout history daemonset <daemonset-name>
```

This returns a list of DaemonSet revisions:

```shell
daemonsets "<daemonset-name>"
REVISION CHANGE-CAUSE
1 ...
2 ...
...
```

* Change cause is copied from DaemonSet annotation `kubernetes.io/change-cause`
to its revisions upon creation. You may specify `--record=true` in `kubectl`
to record the command executed in the change cause annotation.

To see the details of a specific revision:

```shell
kubectl rollout history daemonset <daemonset-name> --revision=1
```

This returns the details of that revision:

```shell
daemonsets "<daemonset-name>" with revision #1
Pod Template:
Labels: foo=bar
Containers:
app:
Image: ...
Port: ...
Environment: ...
Mounts: ...
Volumes: ...
```

### Step 2: Roll back to a specific revision

```shell
# Specify the revision number you get from Step 1 in --to-revision
kubectl rollout undo daemonset <daemonset-name> --to-revision=<revision>
```

If it succeeds, the command returns:

```shell
daemonset "<daemonset-name>" rolled back
```

If `--to-revision` flag is not specified, the last revision will be picked.

### Step 3: Watch the progress of the DaemonSet rollback

`kubectl rollout undo daemonset` tells the server to start rolling back the
DaemonSet. The real rollback is done asynchronously on the server side.

To watch the progress of the rollback:

```shell
kubectl rollout status ds/<daemonset-name>
```

When the rollback is complete, the output is similar to this:

```shell
daemon set "<daemonset-name>" successfully rolled out
```

{% endcapture %}


{% capture discussion %}

## Understanding DaemonSet Revisions

In the previous `kubectl rollout history` step, you got a list of DaemonSet
revisions. Each revision is stored in a resource named `ControllerRevision`.
`ControllerRevision` is a resource only available in Kubernetes release 1.7 or
later.

To see what is stored in each revision, find the DaemonSet revision raw
resources:

```shell
kubectl get controllerrevision -l <daemonset-selector-key>=<daemonset-selector-value>
```

This returns a list of `ControllerRevisions`:

```shell
NAME CONTROLLER REVISION AGE
<daemonset-name>-<revision-hash> DaemonSet/<daemonset-name> 1 1h
<daemonset-name>-<revision-hash> DaemonSet/<daemonset-name> 2 1h
```

Each `ControllerRevision` stores the annotations and template of a DaemonSet
revision.

`kubectl rollout undo` takes a specific `ControllerRevision` and replaces
DaemonSet template with the template stored in the `ControllerRevision`.
`kubectl rollout undo` is equivalent to updating DaemonSet template to a
previous revision through other commands, such as `kubectl edit` or `kubectl
apply`.

Note that DaemonSet revisions only roll forward. That is to say, after a
rollback is complete, the revision number (`.revision` field) of the
`ControllerRevision` being rolled back to will advance. For example, if you
have revision 1 and 2 in the system, and roll back from revision 2 to revision
1, the `ControllerRevision` with `.revision: 1` will become `.revision: 3`.

## Troubleshooting

* See [troubleshooting DaemonSet rolling
update](/docs/tasks/manage-daemon/update-daemon-set/#troubleshooting)

{% endcapture %}

{% include templates/task.md %}
52 changes: 13 additions & 39 deletions docs/tasks/manage-daemon/update-daemon-set.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,46 +33,18 @@ DaemonSet has two update strategy types :
DaemonSet template, old DaemonSet pods will be killed, and new DaemonSet pods
will be created automatically, in a controlled fashion.

## Limitations

* DaemonSet rollout history is not supported yet.
* DaemonSet rollback is not directly supported in `kubectl` yet. You can rollback
by updating DaemonSet template to match the previous version.

## Caveat: Updating DaemonSet created from Kubernetes version 1.5 or before

If you try to rolling update a DaemonSet that was created from Kubernetes
If you try a rolling update on a DaemonSet that was created from Kubernetes
version 1.5 or before, a rollout will be triggered when you *first* change the
DaemonSet update strategy to `RollingUpdate`, even when DaemonSet template isn't
modified. All existing DaemonSet pods will be restarted.

To avoid this restart, first find the DaemonSet's current
`.spec.templateGeneration`:

```shell{% raw %}
kubectl get ds/<daemonset-name> -o go-template='{{.spec.templateGeneration}}{{"\n"}}'
{% endraw %}```
The output should be a number *N*. If the output shows `<no value>`, N = 0.
DaemonSet update strategy to `RollingUpdate`, no matter if DaemonSet template is
modified or not. If the DaemonSet template is not changed, all existing DaemonSet
pods will be restarted (deleted and created).

Then, simply label the existing DaemonSet pods with `pod-template-generation=<N>`
before changing DaemonSet `.spec.updateStrategy` to `RollingUpdate`:
Therefore, make sure you want to trigger a rollout before you first switch the
strategy to `RollingUpdate`.

```shell
# Replace N with DaemonSet `.spec.templateGeneration`
# Only run this on pods created from current DaemonSet template
kubectl label pods -l <daemonset-selector-key>=<daemonset-selector-value> pod-template-generation=<N>
```

This tells the DaemonSet that the labeled DaemonSet pods are created from current
DaemonSet template. Therefore, you should only run this command to existing DaemonSet
pods that are generated from current DaemonSet template.

Note that you only need to do this when you first change the update strategy of
a DaemonSet created from Kubernetes version 1.5 or before to `RollingUpdate`,
but don't want to update its template and start a new rollout yet.

## Setting DaemonSet update strategy for rolling update
## Performing a Rolling Update

To enable the rolling update feature of a DaemonSet, you must set its
`.spec.updateStrategy.type` to `RollingUpdate`.
Expand All @@ -87,15 +59,15 @@ First, check the update strategy of your DaemonSet, and make sure it's set to
RollingUpdate:

```shell{% raw %}
kubectl get ds/<daemonset-name> -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'{% endraw %}
```
kubectl get ds/<daemonset-name> -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'
``` {% endraw %}
If you haven't created the DaemonSet in the system, check your DaemonSet
manifest with the following command instead:
```shell{% raw %}
kubectl create -f ds.yaml --dry-run -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'
{% endraw %}```
``` {% endraw %}
The output from both commands should be:
Expand Down Expand Up @@ -224,7 +196,9 @@ progress.

{% capture whatsnext %}

*TODO: Link to "Task: Creating a DaemonSet to adopt existing DaemonSet pods"*
* See [Task: Performing a rollback on a
DaemonSet](/docs/tasks/manage-daemon/rollback-daemon-set/)
* *TODO: Link to "Task: Creating a DaemonSet to adopt existing DaemonSet pods"*

{% endcapture %}

Expand Down

0 comments on commit 4871d62

Please sign in to comment.