Adds a task for HA etcd kubeadm #8301

chuckha · 2018-05-03T18:25:40Z

Signed-off-by: Chuck Ha ha.chuck@gmail.com

chuckha · 2018-05-03T18:26:49Z

/cc @timothysc and @kubernetes/sig-cluster-lifecycle-pr-reviews

k8s-ci-robot · 2018-05-03T18:26:56Z

@chuckha: Reiterating the mentions to trigger a notification:
@kubernetes/sig-cluster-lifecycle-pr-reviews

In response to this:

/cc @timothysc and @kubernetes/sig-cluster-lifecycle-pr-reviews

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

chuckha · 2018-05-03T18:27:49Z

also cc @Bradamant3 (I'll take care of the hugo transition later, I want to get some eyes on this for the time being)

k8sio-netlify-preview-bot · 2018-05-03T18:28:25Z

Deploy preview for kubernetes-io-master-staging ready!

Built with commit fdb3bc0

https://deploy-preview-8301--kubernetes-io-master-staging.netlify.com

detiber

Outside of updating the SANs for the generated etcd certificates, this looks really good!

detiber · 2018-05-03T19:02:35Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+    serverCertSANs:
+    - "${HOST1}"
+    - "${HOST2}"
+    - "${HOST3}"


Since you are generating unique certs on each etcd host using the copied ca cert, this should probably contain ${CURRENT_HOST} only.

Good catch, this is leftover from my lazy version :)

detiber · 2018-05-03T19:04:28Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+    peerCertSANs:
+    - "${HOST1}"
+    - "${HOST2}"
+    - "${HOST3}"


Bradamant3 · 2018-05-04T00:55:51Z

/assign

chulkilee · 2018-05-04T04:53:05Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+
+### Certificate Authority
+
+Create a CA for the etcd certs. This only has to be done on one node.


Is it possible to use CA generated outside of kubeadm? It looks like if /etc/kubernetes/pki/etcd/ca.{crt,key} exists, kubeadm does not try to create it. Then could we mention that too?

I think this would be useful when using other tools with kubeadm

I think that's worth mentioning. I'll add some copy. Thanks for the suggestion.

chuckha · 2018-05-04T14:18:42Z

_data/tasks.yml

@@ -166,7 +166,7 @@ toc:
  - docs/tasks/administer-cluster/reserve-compute-resources.md
  - docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods.md
  - docs/tasks/administer-cluster/declare-network-policy.md
-  - docs/tasks/administer-cluster/kms-provider.md      
+  - docs/tasks/administer-cluster/kms-provider.md


editor automation 😪

jamiehannaford

looks good! Just a few questions

jamiehannaford · 2018-05-07T09:31:14Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+* Three hosts that can talk to each other over default etcd ports (2379 for
+  clients and 2380 for peers).
+* The `kubeadm` binary installed on each host.
+* SSH or similar to one host and from that host, SSH needs to work to both other


Could we clarify the wording here?

Yeah, I struggled with this one and it shows. Basically, I want to convey that the users can copy files from host A to hosts B and C. scp would be the easiest from a command line perspective...how about something like "SSH between hosts"? The only thing is that this is not really a requirement for the system itself and just to copy certs around. I'll mull on this for a little bit. Happy to listen to suggestions.

jamiehannaford · 2018-05-07T09:31:36Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+* SSH or similar to one host and from that host, SSH needs to work to both other
+  hosts (to copy certificate files around).
+* Docker installed on every host.
+* The `kubelet` installed and running with configuration to read static


maybe we could link them to our installation guide?

+1, it's part of the main instruction so just linking to a block pre-req would be ideal.

good call 👍

jamiehannaford · 2018-05-07T09:32:14Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+section.
+
+```
+export HOST1= # example: 10.0.0.1


could we use dummy vals for this? I think it helps users get an idea of what they need to replace with?

jamiehannaford · 2018-05-07T09:33:14Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+1. `/etc/kubernetes/pki/etcd/ca.crt`
+2. `/etc/kubernetes/pki/etcd/ca.key`
+
+Copy these certs to the other two nodes and put them in `/etc/kubernetes/pki/etcd/`.


do you think it's a good idea to output the commands people can run so they can just copy/paste? I find it lowers the cognitive burden for beginners

I'm going to provide commands for users using ubuntu and we can discuss how you think that looks.

timothysc

I pretty much agree with @jamiehannaford 's comments.

Please address and we'll get this merged.

timothysc · 2018-05-07T15:22:59Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+* SSH or similar to one host and from that host, SSH needs to work to both other
+  hosts (to copy certificate files around).
+* Docker installed on every host.
+* The `kubelet` installed and running with configuration to read static


+1, it's part of the main instruction so just linking to a block pre-req would be ideal.

timothysc · 2018-05-07T15:23:39Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+1. `/etc/kubernetes/pki/etcd/ca.crt`
+2. `/etc/kubernetes/pki/etcd/ca.key`
+
+Copy these certs to the other two nodes and put them in `/etc/kubernetes/pki/etcd/`.


chulkilee · 2018-05-07T16:40:44Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+{% endcapture %}
+
+{% capture prerequisites %}
+* Three hosts that can talk to each other over default etcd ports (2379 for


etcd ports can be configured via kubeadmcfg.yaml so it would be better if this wording reflects that.

chuckha · 2018-05-07T18:58:49Z

@timothysc @jamiehannaford Changes in, would love some feedback, especially on the swizzling bit.

luxas

Great work @chuckha 👏!
Thanks a lot, this is a crucial piece of docs we really need.
Some comments, after those are addressed I want to merge this

luxas · 2018-05-09T15:45:24Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+---
+
+{% capture overview %}
+Kubeadm by default will create a self-hosted etcd cluster of one member. This is


It's not self-hosted as per the definition of "self-hosting" we have in the community. I'd say it like this:

By default, kubeadm will run one etcd instance in a Static Pod ran by the kubelet on the master. This is not a highly-available setup in case your master with that one etcd replica goes unavailable.

Updated to

Kubeadm defaults to running a single member etcd cluster in a static pod managed
by the kubelet on the control plane node. This is not a highly-available setup
as the the etcd cluster contains only one member and cannot sustain any members
becoming unavailable.

luxas · 2018-05-09T15:46:10Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+  document assumes these default ports. However, they are configurable through
+  the kubeadm config file.
+* Each host must [have docker, kubelet, and kubeadm installed][toolbox].
+* SSH between hosts is required for certificate copying.


Could you say you need some infrastructure to copy files around instead of saying SSH is the only way to move files around?

yeah, I've been struggling with this. I think giving ssh as an example is helpful so I came up with

Some infrastructure to copy files between hosts (e.g., ssh).

luxas · 2018-05-09T15:47:39Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+    peerCertSANs:
+    - "${CURRENT_HOST}"
+    extraArgs:
+        initial-cluster-token: etcd-cluster-1


What is this used for?

Ah, great catch. We don't even need this line as it has a default value and we are not spinning up multiple clusters. Here is the relevant documentation:

If spinning up multiple clusters (or creating and destroying a single cluster) with same configuration for testing purpose, it is highly recommended that each cluster is given a unique initial-cluster-token. By doing this, etcd can generate unique cluster IDs and member IDs for the clusters even if they otherwise have the exact same configuration. This can protect etcd from cross-cluster-interaction, which might corrupt the clusters.

luxas · 2018-05-09T15:48:06Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+
+{% capture steps %}
+
+## Kubeadm Configuration File


Create a Configuration File for kubeadm to consume

luxas · 2018-05-09T15:48:52Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+
+## Kubeadm Configuration File
+
+Create a kubeadm configuration file on each host that looks something like this:


something like what's scripted below.?

luxas · 2018-05-09T15:54:06Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+```
+$ kubeadm alpha phase certs etcd-server --config=kubeadmcfg.yaml
+$ kubeadm alpha phase certs etcd-peer --config=kubeadmcfg.yaml
+$ kubeadm alpha phase certs etcd-healthcheck-client --config=kubeadmcfg.yaml


If this is a client cert, I'd rephrase the title to be "unique certs to be generated per peer" or similar

luxas · 2018-05-09T15:54:37Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+`kubeadm` command to generate a static manifest for etcd.
+
+```
+kubeadm alpha phase etcd local --config=kubeadmcfg.yaml


Every time I see alpha here I really look forward to the phases graduation 😄

luxas · 2018-05-09T15:57:45Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+
+## Optional: Check the cluster health
+
+### Get `etcdctl`


Could we use the docker image here instead so we don't have to curl/wget etc. instead just

docker run -it --net host -v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:v3.2.14 etcdctl

makes sense?

hah yes, that is definitely an easier approach.

luxas · 2018-05-09T15:57:56Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+### Check the health
+
+```
+./etcd-v3.1.13-linux-amd64/etcdctl --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --ca-file /etc/kubernetes/pki/etcd/ca.crt --endpoints https://${HOST1}:2379 cluster-health


show successful output?

luxas · 2018-05-09T15:58:06Z

docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

+{% endcapture %}
+
+{% capture whatsnext %}
+* Learn how to use this HA etcd cluster as an external etcd  when bootstrapping


Add link to the next chapter here?

timothysc

/approve

please address @luxas comments and we'll get this merged.

chuckha · 2018-05-09T22:47:48Z

oh right. jekyll got switched out for hugo. Also updating that.

Signed-off-by: Chuck Ha <ha.chuck@gmail.com>

chuckha · 2018-05-09T23:14:16Z

@luxas I changed the content significantly and added new content taking your suggestions into account. I'm better about certs & security here.

timothysc

/lgtm
/approve

If folks have other edits we can do them as a knock-on PR.

k8s-ci-robot · 2018-05-11T15:05:04Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: timothysc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Chuck Ha <ha.chuck@gmail.com>

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 3, 2018

k8s-ci-robot requested review from tengqm and zhangxiaoyu-zidif May 3, 2018 18:25

k8s-ci-robot added the sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. label May 3, 2018

detiber reviewed May 3, 2018

View reviewed changes

k8s-ci-robot assigned Bradamant3 May 4, 2018

chulkilee reviewed May 4, 2018

View reviewed changes

chuckha force-pushed the ha-etcd branch from 9175b8e to ffe6714 Compare May 4, 2018 14:17

chuckha commented May 4, 2018

View reviewed changes

jamiehannaford reviewed May 7, 2018

View reviewed changes

timothysc suggested changes May 7, 2018

View reviewed changes

chulkilee reviewed May 7, 2018

View reviewed changes

chuckha force-pushed the ha-etcd branch 2 times, most recently from 63776df to 3e7a600 Compare May 7, 2018 18:56

chulkilee mentioned this pull request May 8, 2018

Checklist for kubeadm join --control-plane implementation kubernetes/kubeadm#751

Closed

11 tasks

luxas reviewed May 9, 2018

View reviewed changes

timothysc approved these changes May 9, 2018

View reviewed changes

timothysc added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 9, 2018

chuckha force-pushed the ha-etcd branch from 3e7a600 to 6e75644 Compare May 9, 2018 23:10

Adds a task for ha etcd with kubeadm

fdb3bc0

Signed-off-by: Chuck Ha <ha.chuck@gmail.com>

chuckha force-pushed the ha-etcd branch from 6e75644 to fdb3bc0 Compare May 9, 2018 23:12

k8s-ci-robot assigned timothysc May 11, 2018

timothysc approved these changes May 11, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 11, 2018

k8s-ci-robot merged commit 9702579 into kubernetes:master May 11, 2018

mbert pushed a commit to mbert/website that referenced this pull request May 15, 2018

Adds a task for ha etcd with kubeadm (kubernetes#8301)

d009ae1

Signed-off-by: Chuck Ha <ha.chuck@gmail.com>


		### Certificate Authority

		Create a CA for the etcd certs. This only has to be done on one node.


		## Kubeadm Configuration File

		Create a kubeadm configuration file on each host that looks something like this:

Adds a task for HA etcd kubeadm #8301

Adds a task for HA etcd kubeadm #8301

Conversation

chuckha commented May 3, 2018

chuckha commented May 3, 2018

k8s-ci-robot commented May 3, 2018

chuckha commented May 3, 2018

k8sio-netlify-preview-bot commented May 3, 2018 • edited Loading

detiber left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bradamant3 commented May 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamiehannaford left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chuckha May 7, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothysc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chuckha commented May 7, 2018

luxas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothysc left a comment

Choose a reason for hiding this comment

chuckha commented May 9, 2018

chuckha commented May 9, 2018

timothysc left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented May 11, 2018

k8sio-netlify-preview-bot commented May 3, 2018 •

edited

Loading

chuckha May 7, 2018 •

edited

Loading