Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve kubeadm documentation #6645

Merged
merged 1 commit into from
Dec 11, 2017
Merged

Improve kubeadm documentation #6645

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions docs/reference/setup-tools/kubeadm/kubeadm-alpha.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ and behind the scene both use the same code.

## kubeadm alpha phase preflight {#cmd-phase-preflight}

You can execute preflight checks both for he tmaster node, like in `kubeadm init`, or for the worker node
You can execute preflight checks both for the master node, like in `kubeadm init`, or for the worker node
like in `kubeadm join`.

{% capture preflight_master %}
Expand All @@ -46,7 +46,7 @@ like in `kubeadm join`.

## kubeadm alpha phase certs {#cmd-phase-certs}

You can create all required certificates with the `all` sub command or selectively create certificates.
You can create all required certificates with the `all` subcommand or selectively create certificates.

{% capture certs_all %}
{% include_relative generated/kubeadm_alpha_phase_certs_all.md %}
Expand Down Expand Up @@ -84,7 +84,8 @@ You can create all required certificates with the `all` sub command or selective

## kubeadm alpha phase kubeconfig {#cmd-phase-kubeconfig}

You can create all required kubeconfig files with the `all` sub command, or selectively create the files. Additionally, the `user` sub command supports the creation of kubeconfig files for additional users.
You can create all required kubeconfig files with the `all` subcommand, or selectively create the files.
Additionally, the `user` subcommand supports the creation of kubeconfig files for additional users.

{% capture kubeconfig_all %}
{% include_relative generated/kubeadm_alpha_phase_kubeconfig_all.md %}
Expand Down Expand Up @@ -118,7 +119,7 @@ You can create all required kubeconfig files with the `all` sub command, or sele

## kubeadm alpha phase controlplane {#cmd-phase-controlplane}

You can create all required static pod files for control plane components with the `all` sub command,
You can create all required static Pod files for the control plane components with the `all` subcommand,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see why you are capitalizing Pod all over unless it is that way in the rest of the docs.?.?

/cc @Bradamant3

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@timothysc It's the correct convention, yes.

or selectively create the files.

{% capture controlplane_all %}
Expand All @@ -145,7 +146,7 @@ or selectively create the files.

## kubeadm alpha phase etcd {#cmd-phase-etcd}

Use the following command to create a self-hosted, local etcd instance based on a static pod file.
Use the following command to create a self-hosted, local etcd instance based on a static Pod file.

{% capture etcd-local %}
{% include_relative generated/kubeadm_alpha_phase_etcd_local.md %}
Expand All @@ -159,7 +160,7 @@ Use the following command to create a self-hosted, local etcd instance based on

## kubeadm alpha phase mark-master {#cmd-phase-mark-master}

Use the following command to label the node with the `node-role.kubernetes.io/master` taint.
Use the following command to label and taint the node with the `node-role.kubernetes.io/master=""` key-value pair.

{% capture mark-master %}
{% include_relative generated/kubeadm_alpha_phase_mark-master.md %}
Expand All @@ -174,7 +175,7 @@ Use the following command to label the node with the `node-role.kubernetes.io/ma
## kubeadm alpha phase bootstrap-token {#cmd-phase-bootstrap-token}

Use the following actions to fully configure bootstrap tokens.
You can fully configure bootstrap tokens with the `all` sub command,
You can fully configure bootstrap tokens with the `all` subcommand,
or selectively configure single elements.

{% capture bootstrap-token_all %}
Expand Down Expand Up @@ -205,8 +206,8 @@ or selectively configure single elements.

## kubeadm alpha phase upload-config {#cmd-phase-upload-config}

You can use this command to upload the configuration of your cluster. Alternatively, you
can use [kubeadm config](kubeadm-config.md).
You can use this command to upload the kubeadm configuration to your cluster.
Alternatively, you can use [kubeadm config](kubeadm-config.md).

{% capture upload-config %}
{% include_relative generated/kubeadm_alpha_phase_upload-config.md %}
Expand All @@ -220,7 +221,7 @@ can use [kubeadm config](kubeadm-config.md).

## kubeadm alpha phase addon {#cmd-phase-addon}

You can install all the available addons with the `all` sub command, or
You can install all the available addons with the `all` subcommand, or
install them selectively.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add something related to CoreOS?
Add something related to SupportIPVSProxyMode?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be CoreOS-specific?
We don't support that feature gate anymore

{% capture addon-all %}
Expand Down Expand Up @@ -258,4 +259,5 @@ install them selectively.

## What's next
* [kubeadm init](kubeadm-init.md) to bootstrap a Kubernetes master node
* [kubeadm join](kubeadm-join.md) to connect a node to the cluster
* [kubeadm reset](kubeadm-reset.md) to revert any changes made to this host by `kubeadm init` or `kubeadm join`
7 changes: 5 additions & 2 deletions docs/reference/setup-tools/kubeadm/kubeadm-config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,12 @@ title: kubeadm config
---
{% capture overview %}
Beginning with v1.8.0, kubeadm uploads the configuration of your cluster to a ConfigMap called
`kubeadm-config` in the `kube-system` namespace, and later reads the ConfigMap when upgrading. This enables correct configuration of system components, and provides a seamless user experience.
`kubeadm-config` in the `kube-system` namespace, and later reads the ConfigMap when upgrading.
This enables correct configuration of system components, and provides a seamless user experience.

You can use `kubeadm config view` to view the ConfigMap. If you initialized your cluster using kubeadm v1.7.x or lower, you can use `kubeadm config upload` to create a ConfigMap before you execute `kubeadm upgrade`.
You can execute `kubeadm config view` to view the ConfigMap. If you initialized your cluster using
kubeadm v1.7.x or lower, you must use `kubeadm config upload` to create the ConfigMap before you
may use `kubeadm upgrade`.

{% endcapture %}

Expand Down
89 changes: 49 additions & 40 deletions docs/reference/setup-tools/kubeadm/kubeadm-init.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,9 @@ following steps:
considered errors and will exit kubeadm until the problem is corrected or the
user specifies `--skip-preflight-checks`.

1. Generates the token that additional nodes can use to register
themselves with the master in the future. Optionally, the user can provide a
token via `--token`, as described in the
[kubeadm token](kubeadm-token.md) docs.

1. Generates a self-signed CA to provision identities for each component
(including nodes) in the cluster. It also generates client certificates to
be used by various components. If the user has provided their own CA by
dropping it in the cert directory configured via `--cert-dir`
1. Generates a self-signed CA (or using an existing one if provided) to set up
identities for each component in the cluster. If the user has provided their
own CA cert and/or key by dropping it in the cert directory configured via `--cert-dir`
(`/etc/kubernetes/pki` by default) this step is skipped as described in the
[Using custom certificates](#custom-certificates) document.

Expand All @@ -44,11 +38,16 @@ following steps:
Static Pod manifests are written to `/etc/kubernetes/manifests`; the kubelet
watches this directory for Pods to create on startup.

Once control plane Pods are up and running, the kubeadm init sequence can continue.
Once control plane Pods are up and running, the `kubeadm init` sequence can continue.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add something about dynamic kubelet configuration?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure -- feel free to! I was just improving the existing docs, we might wanna add some new, shiny v1.9 stuff as well


1. Apply labels and taints to the master node so that no additional workloads will
run there.

1. Generates the token that additional nodes can use to register
themselves with the master in the future. Optionally, the user can provide a
token via `--token`, as described in the
[kubeadm token](kubeadm-token.md) docs.

1. Makes all the necessary configurations for allowing node joining with the
[Bootstrap Tokens](/docs/admin/bootstrap-tokens/) and
[TLS Bootstrap](/docs/admin/kubelet-tls-bootstrapping/)
Expand All @@ -57,9 +56,9 @@ following steps:
- Write a ConfigMap for making available all the information required
for joining, and set up related RBAC access rules.

- Ensure access to the CSR signing API for bootstrap tokens.
- Let Bootstrap Tokens access the CSR signing API.

- Configure auto approval for new CSR requests.
- Configure auto-approval for new CSR requests.

See [kubeadm join](kubeadm-join.md) for additional info.

Expand Down Expand Up @@ -145,7 +144,7 @@ will need to look like this:
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
apiServerExtraArgs:
feature-gates: APIResponseCompression=true
feature-gates: APIResponseCompression=true
```

To customize the scheduler or controller-manager, use `schedulerExtraArgs` and `controllerManagerExtraArgs` respectively.
Expand All @@ -159,7 +158,7 @@ More information on custom arguments can be found here:

By default, kubeadm pulls images from `gcr.io/google_containers`, unless
the requested Kubernetes version is a CI version. In this case,
`gcr.io/kubernetes-ci-image` is used.
`gcr.io/kubernetes-ci-images` is used.

You can override this behavior by using [kubeadm with a configuration file](#config-file).
Allowed customization are:
Expand All @@ -185,11 +184,16 @@ use case. This means you can, for example, copy an existing CA into `/etc/kubern
and `/etc/kubernetes/pki/ca.key`, and kubeadm will use this CA for signing the rest
of the certs.

#### External CA mode {#external-ca-mode}

It is also possible to provide just the `ca.crt` file and not the
`ca.key` file (this is only available for the root CA file, not other cert pairs).
If all other certificates and kubeconfig files are in place, kubeadm recognizes
this condition and activates the "ExternalCA" mode, which also
implies the CSR signer controller in the controller manager won't be started.
this condition and activates the "External CA" mode. kubeadm will proceed without the
CA key on disk.

Instead, run the controller-manager standalone with `--controllers=csrsigner` and
point to the CA certificate and key.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this is correct.
"Please note that in "External CA" mode, the csrsigning controller will be disabled."

Copy link
Member Author

@luxas luxas Dec 11, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct... or what do you mean?


### Managing the kubeadm drop-in file for the kubelet {#kubelet-drop-in}

Expand Down Expand Up @@ -241,7 +245,7 @@ Here's a breakdown of what/why:
`https://{node-ip}:10250/stats/`. If you want to enable cAdvisor to listen on a
wide-open port, run:

```
```bash
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't seem necessary

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's helpful.

sed -e "/cadvisor-port=0/d" -i /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl daemon-reload
systemctl restart kubelet
Expand All @@ -252,12 +256,11 @@ Here's a breakdown of what/why:

### Use kubeadm with other CRI runtimes

Since the [Kubernetes 1.6 release](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#node-components-1),
Kubernetes has started using CRI as the default container runtime.
The container runtime used by kubeadm is Docker, which is enabled through the built-in
`dockershim` in `kubelet` mechanisms.
Since v1.6.0, Kubernetes has enabled the use of CRI, Container Runtime Interface, by default.
The container runtime used by default is Docker, which is enabled through the built-in
`dockershim` CRI implementation inside of the `kubelet`.

Other CRI-based runtimes include::
Other CRI-based runtimes include:

- [cri-o](https://github.com/kubernetes-incubator/cri-o)
- [frakti](https://github.com/kubernetes/frakti)
Expand All @@ -273,15 +276,18 @@ these two additional steps:
`RUNTIME_ENDPOINT` to your own value like `/var/run/{your_runtime}.sock`:

```shell
cat > /etc/systemd/system/kubelet.service.d/20-cri.conf <<EOF
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=$RUNTIME_ENDPOINT --feature-gates=AllAlpha=true"
cat > /etc/systemd/system/kubelet.service.d/20-cri.conf <<EOF
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=$RUNTIME_ENDPOINT"
EOF
systemctl daemon-reload
systemctl daemon-reload
```

Now `kubelet` is ready to use the specified CRI runtime, and you can continue
with the `kubeadm init` and `kubeadm join` workflow to deploy Kubernetes cluster.

You may also want to set `--cri-socket` to `kubeadm init` and `kubeadm reset` when
using an external CRI implementation.

### Using internal IPs in your cluster

In order to set up a cluster where the master and worker nodes communicate with internal IP addresses (instead of public ones), execute following steps.
Expand All @@ -296,7 +302,8 @@ In order to set up a cluster where the master and worker nodes communicate with

3. Finally, when you run `kubeadm join`, make sure you provide the private IP of the API server addressed as defined in step 1.

### Self-hosting the Kubernetes control plane {#self-hosting}
### Self-hosting the Kubernetes control plane {#self-hosting}

As of 1.8, you can experimentally create a _self-hosted_ Kubernetes control
plane. This means that key components such as the API server, controller
manager, and scheduler run as [DaemonSet pods](/docs/concepts/workloads/controllers/daemonset/)
Expand All @@ -314,7 +321,7 @@ flag to `kubeadm init`.
#### Caveats

Self-hosting in 1.8 has some important limitations. In particular, a
self-hosted cluster cannot recover from a reboot of the master node
self-hosted cluster _cannot recover from a reboot of the master node_
without manual intervention. This and other limitations are expected to be
resolved before self-hosting graduates from alpha.

Expand All @@ -331,8 +338,10 @@ In kubeadm 1.8, the self-hosted portion of the control plane does not include et
which still runs as a static Pod.

#### Process

The self-hosting bootstrap process is documented in the [kubeadm design
document](https://github.com/kubernetes/kubeadm/blob/master/docs/design/design_v1.9.md#optional-self-hosting).

In summary, `kubeadm init --feature-gates=SelfHosting=true` works as follows:

1. Waits for this bootstrap static control plane to be running and
Expand All @@ -359,23 +368,23 @@ This process (steps 3-6) can also be triggered with `kubeadm phase selfhosting c

For running kubeadm without an internet connection you have to pre-pull the required master images for the version of choice:

| Image Name | v1.8 release branch version | v1.9 release branch version
|---|---|---|
| gcr.io/google_containers/kube-apiserver-${ARCH} | v1.8.x | v1.9.x
| gcr.io/google_containers/kube-controller-manager-${ARCH} | v1.8.x | v1.9.x
| gcr.io/google_containers/kube-scheduler-${ARCH} | v1.8.x | v1.9.x
| gcr.io/google_containers/kube-proxy-${ARCH} | v1.8.x | v1.9.x
| gcr.io/google_containers/etcd-${ARCH} | 3.0.17 | 3.1.10
| gcr.io/google_containers/pause-${ARCH} | 3.0 | 3.0
| gcr.io/google_containers/k8s-dns-sidecar-${ARCH} | 1.14.5 | 1.14.7
| gcr.io/google_containers/k8s-dns-kube-dns-${ARCH} | 1.14.5 | 1.14.7
| gcr.io/google_containers/k8s-dns-dnsmasq-nanny-${ARCH} | 1.14.5 | 1.14.7
| Image Name | v1.8 release branch version | v1.9 release branch version |
|----------------------------------------------------------|-----------------------------|-----------------------------|
| gcr.io/google_containers/kube-apiserver-${ARCH} | v1.8.x | v1.9.x |
| gcr.io/google_containers/kube-controller-manager-${ARCH} | v1.8.x | v1.9.x |
| gcr.io/google_containers/kube-scheduler-${ARCH} | v1.8.x | v1.9.x |
| gcr.io/google_containers/kube-proxy-${ARCH} | v1.8.x | v1.9.x |
| gcr.io/google_containers/etcd-${ARCH} | 3.0.17 | 3.1.10 |
| gcr.io/google_containers/pause-${ARCH} | 3.0 | 3.0 |
| gcr.io/google_containers/k8s-dns-sidecar-${ARCH} | 1.14.5 | 1.14.7 |
| gcr.io/google_containers/k8s-dns-kube-dns-${ARCH} | 1.14.5 | 1.14.7 |
| gcr.io/google_containers/k8s-dns-dnsmasq-nanny-${ARCH} | 1.14.5 | 1.14.7 |

Here `v1.8.x` means the "latest patch release of the v1.8 branch".

`${ARCH}` can be one of: `amd64`, `arm`, `arm64`, `ppc64le` or `s390x`.

If using `--feature-gates=CoreDNS` image `coredns/coredns:0.9.10` is required (instead of `kube-dns` images).
If using `--feature-gates=CoreDNS` image `coredns/coredns:1.0.0` is required (instead of the three `k8s-dns-*` images).

### Automating kubeadm

Expand Down Expand Up @@ -414,4 +423,4 @@ provisioned). For details, see the [kubeadm join](kubeadm-join.md).
* [kubeadm reset](kubeadm-reset.md) to revert any changes made to this host by `kubeadm init` or `kubeadm join`
{% endcapture %}

{% include templates/concept.md %}
{% include templates/concept.md %}
18 changes: 10 additions & 8 deletions docs/reference/setup-tools/kubeadm/kubeadm-join.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ title: kubeadm join
{% capture body %}
{% include_relative generated/kubeadm_join.md %}

### Join workflow
### The joining workflow

`kubeadm join` bootstraps a Kubernetes worker node and joins it to the cluster.
This action consists of the following steps:

Expand All @@ -30,12 +31,14 @@ This action consists of the following steps:
1. Finally, kubeadm configures the local kubelet to connect to the API
server with the definitive identity assigned to the node.

### Discovery
### Discovering what cluster CA to trust

The kubeadm discovery has several options, each with security tradeoffs.
The right method for your environment depends on how you provision nodes and the
security expectations you have about your network and node lifecycles.

#### Token-based discovery with CA pinning

This is the default mode in Kubernetes 1.8 and above. In this mode, kubeadm downloads
the cluster configuration (including root CA) and validates it using the token
as well as validating that the root CA public key matches the provided hash and
Expand All @@ -49,7 +52,7 @@ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outfor

**Example `kubeadm join` command:**

```
```bash
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Syntax-specific formatting is helpful.

kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef 1.2.3.4:6443
```

Expand All @@ -65,9 +68,11 @@ kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert

- The CA hash is not normally known until the master has been provisioned,
which can make it more difficult to build automated provisioning tools that
use kubeadm.
use kubeadm. By generating your CA in beforehand, you may workaround this
limitation though.

#### Token-based discovery without CA pinning

_This was the default in Kubernetes 1.7 and earlier_, but comes with some
important caveats. This mode relies only on the symmetric token to sign
(HMAC-SHA256) the discovery information that establishes the root of trust for
Expand Down Expand Up @@ -119,9 +124,6 @@ using kubeadm.
via your cloud provider or provisioning tool. The information in this file is
not secret, but HTTPS or equivalent is required to ensure its integrity.

- Less convenient to use manually since the file is difficult to copy and paste
between nodes.

### Securing your installation even more {#securing-more}

The defaults for kubeadm may not work for everyone. This section documents how to tighten up a kubeadm installation
Expand Down Expand Up @@ -224,4 +226,4 @@ discoveryTokenUnsafeSkipCAVerification: <bool>
* [kubeadm reset](kubeadm-reset.md) to revert any changes made to this host by `kubeadm init` or `kubeadm join`
{% endcapture %}

{% include templates/concept.md %}
{% include templates/concept.md %}
5 changes: 3 additions & 2 deletions docs/reference/setup-tools/kubeadm/kubeadm-reset.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,16 @@ title: kubeadm reset
{% include_relative generated/kubeadm_reset.md %}

### External etcd clean up

`kubeadm reset` will not delete any etcd data if external etcd is used. This means that if you run `kubeadm init` again using the same etcd endpoints, you will see state from previous clusters.

To wipe etcd data it is recommended you use a client like etcdctl, such as:

```
```bash
etcdctl del "" --prefix
```

See [etcd documentation](https://github.com/coreos/etcd/tree/master/etcdctl) for more information.
See the [etcd documentation](https://github.com/coreos/etcd/tree/master/etcdctl) for more information.
{% endcapture %}

{% capture whatsnext %}
Expand Down
Loading