Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update task for DaemonSet history and rollback #4098

Merged
merged 2 commits into from
Jun 22, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions _data/tasks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,7 @@ toc:
- title: Managing Cluster Daemons
section:
- docs/tasks/manage-daemon/update-daemon-set.md
- docs/tasks/manage-daemon/rollback-daemon-set.md

- title: Managing GPUs
section:
Expand Down
154 changes: 154 additions & 0 deletions docs/tasks/manage-daemon/rollback-daemon-set.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
---
assignees:
- janetkuo
title: Performing a Rollback on a DaemonSet
---

{% capture overview %}

This page shows how to perform a rollback on a DaemonSet.

{% endcapture %}


{% capture prerequisites %}

* The DaemonSet rollout history and DaemonSet rollback features are only
supported in `kubectl` in Kubernetes version 1.7 or later.
* Make sure you know how to [perform a rolling update on a
DaemonSet](/docs/tasks/manage-daemon/update-daemon-set/)

{% endcapture %}


{% capture steps %}

## Performing a Rollback on a DaemonSet

### Step 1: Find the DaemonSet revision you want to roll back to

You can skip this step if you just want to roll back to the last revision.

List all revisions of a DaemonSet:

```shell
kubectl rollout history daemonset <daemonset-name>
```

This returns a list of DaemonSet revisions:

```shell
daemonsets "<daemonset-name>"
REVISION CHANGE-CAUSE
1 ...
2 ...
...
```

* Change cause is copied from DaemonSet annotation `kubernetes.io/change-cause`
to its revisions upon creation. You may specify `--record=true` in `kubectl`
to record the command executed in the change cause annotation.

To see the details of a specific revision:

```shell
kubectl rollout history daemonset <daemonset-name> --revision=1
```

This returns the details of that revision:

```shell
daemonsets "<daemonset-name>" with revision #1
Pod Template:
Labels: foo=bar
Containers:
app:
Image: ...
Port: ...
Environment: ...
Mounts: ...
Volumes: ...
```

### Step 2: Roll back to a specific revision

```shell
# Specify the revision number you get from Step 1 in --to-revision
kubectl rollout undo daemonset <daemonset-name> --to-revision=<revision>
```

If it succeeds, the command returns:

```shell
daemonset "<daemonset-name>" rolled back
```

If `--to-revision` flag is not specified, the last revision will be picked.

### Step 3: Watch the progress of the DaemonSet rollback

`kubectl rollout undo daemonset` tells the server to start rolling back the
DaemonSet. The real rollback is done asynchronously on the server side.

To watch the progress of the rollback:

```shell
kubectl rollout status ds/<daemonset-name>
```

When the rollback is complete, the output is similar to this:

```shell
daemon set "<daemonset-name>" successfully rolled out
```

{% endcapture %}


{% capture discussion %}

## Understanding DaemonSet Revisions

In the previous `kubectl rollout history` step, you got a list of DaemonSet
revisions. Each revision is stored in a resource named `ControllerRevision`.
`ControllerRevision` is a resource only available in Kubernetes release 1.7 or
later.

To see what is stored in each revision, find the DaemonSet revision raw
resources:

```shell
kubectl get controllerrevision -l <daemonset-selector-key>=<daemonset-selector-value>
```

This returns a list of `ControllerRevisions`:

```shell
NAME CONTROLLER REVISION AGE
<daemonset-name>-<revision-hash> DaemonSet/<daemonset-name> 1 1h
<daemonset-name>-<revision-hash> DaemonSet/<daemonset-name> 2 1h
```

Each `ControllerRevision` stores the annotations and template of a DaemonSet
revision.

`kubectl rollout undo` takes a specific `ControllerRevision` and replaces
DaemonSet template with the template stored in the `ControllerRevision`.
`kubectl rollout undo` is equivalent to updating DaemonSet template to a
previous revision through other commands, such as `kubectl edit` or `kubectl
apply`.

Note that DaemonSet revisions only roll forward. That is to say, after a
rollback is complete, the revision number (`.revision` field) of the
`ControllerRevision` being rolled back to will advance. For example, if you
have revision 1 and 2 in the system, and roll back from revision 2 to revision
1, the `ControllerRevision` with `.revision: 1` will become `.revision: 3`.

## Troubleshooting

* See [troubleshooting DaemonSet rolling
update](/docs/tasks/manage-daemon/update-daemon-set/#troubleshooting)

{% endcapture %}

{% include templates/task.md %}
52 changes: 13 additions & 39 deletions docs/tasks/manage-daemon/update-daemon-set.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,46 +33,18 @@ DaemonSet has two update strategy types :
DaemonSet template, old DaemonSet pods will be killed, and new DaemonSet pods
will be created automatically, in a controlled fashion.

## Limitations

* DaemonSet rollout history is not supported yet.
* DaemonSet rollback is not directly supported in `kubectl` yet. You can rollback
by updating DaemonSet template to match the previous version.

## Caveat: Updating DaemonSet created from Kubernetes version 1.5 or before

If you try to rolling update a DaemonSet that was created from Kubernetes
If you try a rolling update on a DaemonSet that was created from Kubernetes
version 1.5 or before, a rollout will be triggered when you *first* change the
DaemonSet update strategy to `RollingUpdate`, even when DaemonSet template isn't
modified. All existing DaemonSet pods will be restarted.

To avoid this restart, first find the DaemonSet's current
`.spec.templateGeneration`:

```shell{% raw %}
kubectl get ds/<daemonset-name> -o go-template='{{.spec.templateGeneration}}{{"\n"}}'
{% endraw %}```

The output should be a number *N*. If the output shows `<no value>`, N = 0.
DaemonSet update strategy to `RollingUpdate`, no matter if DaemonSet template is
modified or not. If the DaemonSet template is not changed, all existing DaemonSet
pods will be restarted (deleted and created).

Then, simply label the existing DaemonSet pods with `pod-template-generation=<N>`
before changing DaemonSet `.spec.updateStrategy` to `RollingUpdate`:
Therefore, make sure you want to trigger a rollout before you first switch the
strategy to `RollingUpdate`.

```shell
# Replace N with DaemonSet `.spec.templateGeneration`
# Only run this on pods created from current DaemonSet template
kubectl label pods -l <daemonset-selector-key>=<daemonset-selector-value> pod-template-generation=<N>
```

This tells the DaemonSet that the labeled DaemonSet pods are created from current
DaemonSet template. Therefore, you should only run this command to existing DaemonSet
pods that are generated from current DaemonSet template.

Note that you only need to do this when you first change the update strategy of
a DaemonSet created from Kubernetes version 1.5 or before to `RollingUpdate`,
but don't want to update its template and start a new rollout yet.

## Setting DaemonSet update strategy for rolling update
## Performing a Rolling Update

To enable the rolling update feature of a DaemonSet, you must set its
`.spec.updateStrategy.type` to `RollingUpdate`.
Expand All @@ -87,15 +59,15 @@ First, check the update strategy of your DaemonSet, and make sure it's set to
RollingUpdate:

```shell{% raw %}
kubectl get ds/<daemonset-name> -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'{% endraw %}
```
kubectl get ds/<daemonset-name> -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'
``` {% endraw %}

If you haven't created the DaemonSet in the system, check your DaemonSet
manifest with the following command instead:

```shell{% raw %}
kubectl create -f ds.yaml --dry-run -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'
{% endraw %}```
``` {% endraw %}

The output from both commands should be:

Expand Down Expand Up @@ -224,7 +196,9 @@ progress.

{% capture whatsnext %}

*TODO: Link to "Task: Creating a DaemonSet to adopt existing DaemonSet pods"*
* See [Task: Performing a rollback on a
DaemonSet](/docs/tasks/manage-daemon/rollback-daemon-set/)
* *TODO: Link to "Task: Creating a DaemonSet to adopt existing DaemonSet pods"*

{% endcapture %}

Expand Down