Skip to content

Commit

Permalink
kubeadm: Improve kubeadm documentation for v1.9
Browse files Browse the repository at this point in the history
  • Loading branch information
luxas committed Dec 11, 2017
1 parent 3944be8 commit 71d7fdb
Show file tree
Hide file tree
Showing 11 changed files with 272 additions and 257 deletions.
22 changes: 12 additions & 10 deletions docs/reference/setup-tools/kubeadm/kubeadm-alpha.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ and behind the scene both use the same code.

## kubeadm alpha phase preflight {#cmd-phase-preflight}

You can execute preflight checks both for he tmaster node, like in `kubeadm init`, or for the worker node
You can execute preflight checks both for the master node, like in `kubeadm init`, or for the worker node
like in `kubeadm join`.

{% capture preflight_master %}
Expand All @@ -46,7 +46,7 @@ like in `kubeadm join`.

## kubeadm alpha phase certs {#cmd-phase-certs}

You can create all required certificates with the `all` sub command or selectively create certificates.
You can create all required certificates with the `all` subcommand or selectively create certificates.

{% capture certs_all %}
{% include_relative generated/kubeadm_alpha_phase_certs_all.md %}
Expand Down Expand Up @@ -84,7 +84,8 @@ You can create all required certificates with the `all` sub command or selective

## kubeadm alpha phase kubeconfig {#cmd-phase-kubeconfig}

You can create all required kubeconfig files with the `all` sub command, or selectively create the files. Additionally, the `user` sub command supports the creation of kubeconfig files for additional users.
You can create all required kubeconfig files with the `all` subcommand, or selectively create the files.
Additionally, the `user` subcommand supports the creation of kubeconfig files for additional users.

{% capture kubeconfig_all %}
{% include_relative generated/kubeadm_alpha_phase_kubeconfig_all.md %}
Expand Down Expand Up @@ -118,7 +119,7 @@ You can create all required kubeconfig files with the `all` sub command, or sele

## kubeadm alpha phase controlplane {#cmd-phase-controlplane}

You can create all required static pod files for control plane components with the `all` sub command,
You can create all required static Pod files for the control plane components with the `all` subcommand,
or selectively create the files.

{% capture controlplane_all %}
Expand All @@ -145,7 +146,7 @@ or selectively create the files.

## kubeadm alpha phase etcd {#cmd-phase-etcd}

Use the following command to create a self-hosted, local etcd instance based on a static pod file.
Use the following command to create a self-hosted, local etcd instance based on a static Pod file.

{% capture etcd-local %}
{% include_relative generated/kubeadm_alpha_phase_etcd_local.md %}
Expand All @@ -159,7 +160,7 @@ Use the following command to create a self-hosted, local etcd instance based on

## kubeadm alpha phase mark-master {#cmd-phase-mark-master}

Use the following command to label the node with the `node-role.kubernetes.io/master` taint.
Use the following command to label and taint the node with the `node-role.kubernetes.io/master=""` key-value pair.

{% capture mark-master %}
{% include_relative generated/kubeadm_alpha_phase_mark-master.md %}
Expand All @@ -174,7 +175,7 @@ Use the following command to label the node with the `node-role.kubernetes.io/ma
## kubeadm alpha phase bootstrap-token {#cmd-phase-bootstrap-token}

Use the following actions to fully configure bootstrap tokens.
You can fully configure bootstrap tokens with the `all` sub command,
You can fully configure bootstrap tokens with the `all` subcommand,
or selectively configure single elements.

{% capture bootstrap-token_all %}
Expand Down Expand Up @@ -205,8 +206,8 @@ or selectively configure single elements.

## kubeadm alpha phase upload-config {#cmd-phase-upload-config}

You can use this command to upload the configuration of your cluster. Alternatively, you
can use [kubeadm config](kubeadm-config.md).
You can use this command to upload the kubeadm configuration to your cluster.
Alternatively, you can use [kubeadm config](kubeadm-config.md).

{% capture upload-config %}
{% include_relative generated/kubeadm_alpha_phase_upload-config.md %}
Expand All @@ -220,7 +221,7 @@ can use [kubeadm config](kubeadm-config.md).

## kubeadm alpha phase addon {#cmd-phase-addon}

You can install all the available addons with the `all` sub command, or
You can install all the available addons with the `all` subcommand, or
install them selectively.

{% capture addon-all %}
Expand Down Expand Up @@ -258,4 +259,5 @@ install them selectively.

## What's next
* [kubeadm init](kubeadm-init.md) to bootstrap a Kubernetes master node
* [kubeadm join](kubeadm-join.md) to connect a node to the cluster
* [kubeadm reset](kubeadm-reset.md) to revert any changes made to this host by `kubeadm init` or `kubeadm join`
7 changes: 5 additions & 2 deletions docs/reference/setup-tools/kubeadm/kubeadm-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,12 @@ title: kubeadm config
---
{% capture overview %}
Beginning with v1.8.0, kubeadm uploads the configuration of your cluster to a ConfigMap called
`kubeadm-config` in the `kube-system` namespace, and later reads the ConfigMap when upgrading. This enables correct configuration of system components, and provides a seamless user experience.
`kubeadm-config` in the `kube-system` namespace, and later reads the ConfigMap when upgrading.
This enables correct configuration of system components, and provides a seamless user experience.

You can use `kubeadm config view` to view the ConfigMap. If you initialized your cluster using kubeadm v1.7.x or lower, you can use `kubeadm config upload` to create a ConfigMap before you execute `kubeadm upgrade`.
You can execute `kubeadm config view` to view the ConfigMap. If you initialized your cluster using
kubeadm v1.7.x or lower, you must use `kubeadm config upload` to create the ConfigMap before you
may use `kubeadm upgrade`.

{% endcapture %}

Expand Down
89 changes: 49 additions & 40 deletions docs/reference/setup-tools/kubeadm/kubeadm-init.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,9 @@ following steps:
considered errors and will exit kubeadm until the problem is corrected or the
user specifies `--skip-preflight-checks`.

1. Generates the token that additional nodes can use to register
themselves with the master in the future. Optionally, the user can provide a
token via `--token`, as described in the
[kubeadm token](kubeadm-token.md) docs.

1. Generates a self-signed CA to provision identities for each component
(including nodes) in the cluster. It also generates client certificates to
be used by various components. If the user has provided their own CA by
dropping it in the cert directory configured via `--cert-dir`
1. Generates a self-signed CA (or using an existing one if provided) to set up
identities for each component in the cluster. If the user has provided their
own CA cert and/or key by dropping it in the cert directory configured via `--cert-dir`
(`/etc/kubernetes/pki` by default) this step is skipped as described in the
[Using custom certificates](#custom-certificates) document.

Expand All @@ -44,11 +38,16 @@ following steps:
Static Pod manifests are written to `/etc/kubernetes/manifests`; the kubelet
watches this directory for Pods to create on startup.

Once control plane Pods are up and running, the kubeadm init sequence can continue.
Once control plane Pods are up and running, the `kubeadm init` sequence can continue.

1. Apply labels and taints to the master node so that no additional workloads will
run there.

1. Generates the token that additional nodes can use to register
themselves with the master in the future. Optionally, the user can provide a
token via `--token`, as described in the
[kubeadm token](kubeadm-token.md) docs.

1. Makes all the necessary configurations for allowing node joining with the
[Bootstrap Tokens](/docs/admin/bootstrap-tokens/) and
[TLS Bootstrap](/docs/admin/kubelet-tls-bootstrapping/)
Expand All @@ -57,9 +56,9 @@ following steps:
- Write a ConfigMap for making available all the information required
for joining, and set up related RBAC access rules.

- Ensure access to the CSR signing API for bootstrap tokens.
- Let Bootstrap Tokens access the CSR signing API.

- Configure auto approval for new CSR requests.
- Configure auto-approval for new CSR requests.

See [kubeadm join](kubeadm-join.md) for additional info.

Expand Down Expand Up @@ -145,7 +144,7 @@ will need to look like this:
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
apiServerExtraArgs:
feature-gates: APIResponseCompression=true
feature-gates: APIResponseCompression=true
```
To customize the scheduler or controller-manager, use `schedulerExtraArgs` and `controllerManagerExtraArgs` respectively.
Expand All @@ -159,7 +158,7 @@ More information on custom arguments can be found here:

By default, kubeadm pulls images from `gcr.io/google_containers`, unless
the requested Kubernetes version is a CI version. In this case,
`gcr.io/kubernetes-ci-image` is used.
`gcr.io/kubernetes-ci-images` is used.

You can override this behavior by using [kubeadm with a configuration file](#config-file).
Allowed customization are:
Expand All @@ -185,11 +184,16 @@ use case. This means you can, for example, copy an existing CA into `/etc/kubern
and `/etc/kubernetes/pki/ca.key`, and kubeadm will use this CA for signing the rest
of the certs.

#### External CA mode {#external-ca-mode}

It is also possible to provide just the `ca.crt` file and not the
`ca.key` file (this is only available for the root CA file, not other cert pairs).
If all other certificates and kubeconfig files are in place, kubeadm recognizes
this condition and activates the "ExternalCA" mode, which also
implies the CSR signer controller in the controller manager won't be started.
this condition and activates the "External CA" mode. kubeadm will proceed without the
CA key on disk.

Instead, run the controller-manager standalone with `--controllers=csrsigner` and
point to the CA certificate and key.

### Managing the kubeadm drop-in file for the kubelet {#kubelet-drop-in}

Expand Down Expand Up @@ -241,7 +245,7 @@ Here's a breakdown of what/why:
`https://{node-ip}:10250/stats/`. If you want to enable cAdvisor to listen on a
wide-open port, run:

```
```bash
sed -e "/cadvisor-port=0/d" -i /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl daemon-reload
systemctl restart kubelet
Expand All @@ -252,12 +256,11 @@ Here's a breakdown of what/why:

### Use kubeadm with other CRI runtimes

Since the [Kubernetes 1.6 release](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#node-components-1),
Kubernetes has started using CRI as the default container runtime.
The container runtime used by kubeadm is Docker, which is enabled through the built-in
`dockershim` in `kubelet` mechanisms.
Since v1.6.0, Kubernetes has enabled the use of CRI, Container Runtime Interface, by default.
The container runtime used by default is Docker, which is enabled through the built-in
`dockershim` CRI implementation inside of the `kubelet`.

Other CRI-based runtimes include::
Other CRI-based runtimes include:

- [cri-o](https://github.com/kubernetes-incubator/cri-o)
- [frakti](https://github.com/kubernetes/frakti)
Expand All @@ -273,15 +276,18 @@ these two additional steps:
`RUNTIME_ENDPOINT` to your own value like `/var/run/{your_runtime}.sock`:

```shell
cat > /etc/systemd/system/kubelet.service.d/20-cri.conf <<EOF
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=$RUNTIME_ENDPOINT --feature-gates=AllAlpha=true"
cat > /etc/systemd/system/kubelet.service.d/20-cri.conf <<EOF
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=$RUNTIME_ENDPOINT"
EOF
systemctl daemon-reload
systemctl daemon-reload
```

Now `kubelet` is ready to use the specified CRI runtime, and you can continue
with the `kubeadm init` and `kubeadm join` workflow to deploy Kubernetes cluster.

You may also want to set `--cri-socket` to `kubeadm init` and `kubeadm reset` when
using an external CRI implementation.

### Using internal IPs in your cluster

In order to set up a cluster where the master and worker nodes communicate with internal IP addresses (instead of public ones), execute following steps.
Expand All @@ -296,7 +302,8 @@ In order to set up a cluster where the master and worker nodes communicate with

3. Finally, when you run `kubeadm join`, make sure you provide the private IP of the API server addressed as defined in step 1.

### Self-hosting the Kubernetes control plane {#self-hosting}
### Self-hosting the Kubernetes control plane {#self-hosting}

As of 1.8, you can experimentally create a _self-hosted_ Kubernetes control
plane. This means that key components such as the API server, controller
manager, and scheduler run as [DaemonSet pods](/docs/concepts/workloads/controllers/daemonset/)
Expand All @@ -314,7 +321,7 @@ flag to `kubeadm init`.
#### Caveats

Self-hosting in 1.8 has some important limitations. In particular, a
self-hosted cluster cannot recover from a reboot of the master node
self-hosted cluster _cannot recover from a reboot of the master node_
without manual intervention. This and other limitations are expected to be
resolved before self-hosting graduates from alpha.

Expand All @@ -331,8 +338,10 @@ In kubeadm 1.8, the self-hosted portion of the control plane does not include et
which still runs as a static Pod.

#### Process

The self-hosting bootstrap process is documented in the [kubeadm design
document](https://github.com/kubernetes/kubeadm/blob/master/docs/design/design_v1.9.md#optional-self-hosting).

In summary, `kubeadm init --feature-gates=SelfHosting=true` works as follows:

1. Waits for this bootstrap static control plane to be running and
Expand All @@ -359,23 +368,23 @@ This process (steps 3-6) can also be triggered with `kubeadm phase selfhosting c

For running kubeadm without an internet connection you have to pre-pull the required master images for the version of choice:

| Image Name | v1.8 release branch version | v1.9 release branch version
|---|---|---|
| gcr.io/google_containers/kube-apiserver-${ARCH} | v1.8.x | v1.9.x
| gcr.io/google_containers/kube-controller-manager-${ARCH} | v1.8.x | v1.9.x
| gcr.io/google_containers/kube-scheduler-${ARCH} | v1.8.x | v1.9.x
| gcr.io/google_containers/kube-proxy-${ARCH} | v1.8.x | v1.9.x
| gcr.io/google_containers/etcd-${ARCH} | 3.0.17 | 3.1.10
| gcr.io/google_containers/pause-${ARCH} | 3.0 | 3.0
| gcr.io/google_containers/k8s-dns-sidecar-${ARCH} | 1.14.5 | 1.14.7
| gcr.io/google_containers/k8s-dns-kube-dns-${ARCH} | 1.14.5 | 1.14.7
| gcr.io/google_containers/k8s-dns-dnsmasq-nanny-${ARCH} | 1.14.5 | 1.14.7
| Image Name | v1.8 release branch version | v1.9 release branch version |
|----------------------------------------------------------|-----------------------------|-----------------------------|
| gcr.io/google_containers/kube-apiserver-${ARCH} | v1.8.x | v1.9.x |
| gcr.io/google_containers/kube-controller-manager-${ARCH} | v1.8.x | v1.9.x |
| gcr.io/google_containers/kube-scheduler-${ARCH} | v1.8.x | v1.9.x |
| gcr.io/google_containers/kube-proxy-${ARCH} | v1.8.x | v1.9.x |
| gcr.io/google_containers/etcd-${ARCH} | 3.0.17 | 3.1.10 |
| gcr.io/google_containers/pause-${ARCH} | 3.0 | 3.0 |
| gcr.io/google_containers/k8s-dns-sidecar-${ARCH} | 1.14.5 | 1.14.7 |
| gcr.io/google_containers/k8s-dns-kube-dns-${ARCH} | 1.14.5 | 1.14.7 |
| gcr.io/google_containers/k8s-dns-dnsmasq-nanny-${ARCH} | 1.14.5 | 1.14.7 |

Here `v1.8.x` means the "latest patch release of the v1.8 branch".

`${ARCH}` can be one of: `amd64`, `arm`, `arm64`, `ppc64le` or `s390x`.

If using `--feature-gates=CoreDNS` image `coredns/coredns:0.9.10` is required (instead of `kube-dns` images).
If using `--feature-gates=CoreDNS` image `coredns/coredns:1.0.0` is required (instead of the three `k8s-dns-*` images).

### Automating kubeadm

Expand Down Expand Up @@ -414,4 +423,4 @@ provisioned). For details, see the [kubeadm join](kubeadm-join.md).
* [kubeadm reset](kubeadm-reset.md) to revert any changes made to this host by `kubeadm init` or `kubeadm join`
{% endcapture %}

{% include templates/concept.md %}
{% include templates/concept.md %}
18 changes: 10 additions & 8 deletions docs/reference/setup-tools/kubeadm/kubeadm-join.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ title: kubeadm join
{% capture body %}
{% include_relative generated/kubeadm_join.md %}

### Join workflow
### The joining workflow

`kubeadm join` bootstraps a Kubernetes worker node and joins it to the cluster.
This action consists of the following steps:

Expand All @@ -30,12 +31,14 @@ This action consists of the following steps:
1. Finally, kubeadm configures the local kubelet to connect to the API
server with the definitive identity assigned to the node.

### Discovery
### Discovering what cluster CA to trust

The kubeadm discovery has several options, each with security tradeoffs.
The right method for your environment depends on how you provision nodes and the
security expectations you have about your network and node lifecycles.

#### Token-based discovery with CA pinning

This is the default mode in Kubernetes 1.8 and above. In this mode, kubeadm downloads
the cluster configuration (including root CA) and validates it using the token
as well as validating that the root CA public key matches the provided hash and
Expand All @@ -49,7 +52,7 @@ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outfor

**Example `kubeadm join` command:**

```
```bash
kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef 1.2.3.4:6443
```

Expand All @@ -65,9 +68,11 @@ kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert

- The CA hash is not normally known until the master has been provisioned,
which can make it more difficult to build automated provisioning tools that
use kubeadm.
use kubeadm. By generating your CA in beforehand, you may workaround this
limitation though.

#### Token-based discovery without CA pinning

_This was the default in Kubernetes 1.7 and earlier_, but comes with some
important caveats. This mode relies only on the symmetric token to sign
(HMAC-SHA256) the discovery information that establishes the root of trust for
Expand Down Expand Up @@ -119,9 +124,6 @@ using kubeadm.
via your cloud provider or provisioning tool. The information in this file is
not secret, but HTTPS or equivalent is required to ensure its integrity.

- Less convenient to use manually since the file is difficult to copy and paste
between nodes.

### Securing your installation even more {#securing-more}

The defaults for kubeadm may not work for everyone. This section documents how to tighten up a kubeadm installation
Expand Down Expand Up @@ -224,4 +226,4 @@ discoveryTokenUnsafeSkipCAVerification: <bool>
* [kubeadm reset](kubeadm-reset.md) to revert any changes made to this host by `kubeadm init` or `kubeadm join`
{% endcapture %}

{% include templates/concept.md %}
{% include templates/concept.md %}
5 changes: 3 additions & 2 deletions docs/reference/setup-tools/kubeadm/kubeadm-reset.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,16 @@ title: kubeadm reset
{% include_relative generated/kubeadm_reset.md %}

### External etcd clean up

`kubeadm reset` will not delete any etcd data if external etcd is used. This means that if you run `kubeadm init` again using the same etcd endpoints, you will see state from previous clusters.

To wipe etcd data it is recommended you use a client like etcdctl, such as:

```
```bash
etcdctl del "" --prefix
```

See [etcd documentation](https://github.com/coreos/etcd/tree/master/etcdctl) for more information.
See the [etcd documentation](https://github.com/coreos/etcd/tree/master/etcdctl) for more information.
{% endcapture %}

{% capture whatsnext %}
Expand Down
Loading

0 comments on commit 71d7fdb

Please sign in to comment.