Add a doc about the kubeadm design and phases implementation #156

luxas · 2017-02-09T15:21:15Z

This document describes the functions of kubeadm and how the phases should be splitted up

@lukemarsden @roberthbailey @mikedanese @errordeveloper @pires @dmmcquay @jbeda

dmmcquay

Edits to make the doc more readable.

dmmcquay · 2017-02-09T18:10:06Z

docs/design/phases.md

@@ -0,0 +1,110 @@
+## Phases in kubeadm
+
+`kubeadm init` does a lot of things. In order to make kubeadm the common building block for Kubernetes we want it to be, each logical phase kubeadm does during `init` should be invokable separately. Here's an explanation what kubeadm does in the different phases:


Suggested edit:

kubeadm init does a lot. In order to make kubeadm the common building block for Kubernetes, each logical phase kubeadm does during init should be invokable separately. Here's an explanation what kubeadm does in these different phases:

Thanks for the grammatical nits, I was in a hurry when writing this so I didn't really pay attention to my wording in this doc, will fix 👍

dmmcquay · 2017-02-09T18:11:11Z

docs/design/phases.md

+
+#### A note on constants / well-known values and paths
+
+Somewhere we have to draw the line what should be configurable, what shouldn't and what should be hard-coded in the binary.


Suggested edit:

We have to draw the line somewhere about what should be configurable, what shouldn't, and what should be hard-coded in the binary.

dmmcquay · 2017-02-09T18:12:48Z

docs/design/phases.md

+Somewhere we have to draw the line what should be configurable, what shouldn't and what should be hard-coded in the binary.
+
+We've decided to make the Kubernetes directory `/etc/kubernetes` a constant in the application, since it is clearly the given path in a lot of places,
+and the most intuitive location. Having that path configurable would for sure confuse readers of a on-top-of-kubeadm-implemented deployment solution.


Suggested edit:

and the most intuitive location. Having that path configurable would confuse readers of an on-top-of-kubeadm-implemented deployment solution.

dmmcquay · 2017-02-09T18:13:18Z

docs/design/phases.md

+
+Somewhere we have to draw the line what should be configurable, what shouldn't and what should be hard-coded in the binary.
+
+We've decided to make the Kubernetes directory `/etc/kubernetes` a constant in the application, since it is clearly the given path in a lot of places,


Suggested edit:

We've decided to make the Kubernetes directory /etc/kubernetes a constant in the application, since it is clearly the given path in a majority of cases,

(I'm not even sure you need to justify the choice of /etc/kubernetes)

dmmcquay · 2017-02-09T18:13:45Z

docs/design/phases.md

+We've decided to make the Kubernetes directory `/etc/kubernetes` a constant in the application, since it is clearly the given path in a lot of places,
+and the most intuitive location. Having that path configurable would for sure confuse readers of a on-top-of-kubeadm-implemented deployment solution.
+
+Further, having the admin/kubelet kubeconfig path would 


This doesn't make sense to me.

yeah, this is just an incomplete thought

dmmcquay · 2017-02-09T18:16:44Z

docs/design/phases.md

+   - ??
+
+If a given certificate and private key pair both exist, the generation will be skipped, and those files will validated and used if they are valid for the use-case.
+This means the user can for example give his own CA which then will be used signing the rest of the certs.


Suggested edit:

This means the user can, for example, give his own CA, which then will be used for signing the rest of the certs.

dmmcquay · 2017-02-09T18:17:59Z

docs/design/phases.md

+
+### Bootstrap the control plane
+
+First, see so there is an etcd cluster somewhere:


Suggested edit:

First, see that there is an etcd cluster somewhere:

Suggested edit (nit-picky, only because it's been called out already):

going just a bit further: "First, determine if there is an etcd cluster available:"

dmmcquay · 2017-02-09T18:18:49Z

docs/design/phases.md

+
+First, see so there is an etcd cluster somewhere:
+ - use an external one if external etcd params are given
+ - spin up one locally if external etcd should not be enabled. It should:


Suggested edit:

spin up one locally if external etcd is not be enabled. It should:

dmmcquay · 2017-02-09T18:20:17Z

docs/design/phases.md

+
+Then kubeadm waits for the control plane to come up as static pods
+
+The master node is patched; the master label is added and the master is tainted.


Suggested edit:

The master node is patched, the master label is added, and the master is tainted.

dmmcquay · 2017-02-09T18:20:47Z

docs/design/phases.md

+ - kube-proxy; the `kube-proxy` ServiceAccount can gets the privileges in the `system:node-proxier` ClusterRole
+ - kube-dns; the `kube-dns` ServiceAccount can gets the privileges in the `system:kube-dns` ClusterRole
+
+In the future; the base layer of configuration for all kubelets in the cluster will be set in this phase.


Suggested edit:

In the future the base layer of configuration for all kubelets in the cluster will be set in this phase.

substitute ; for ,

luxas · 2017-02-09T19:06:46Z

cc @mattymo @v1k0d3n @justinsb @chrislovecnm
You're building tools on top of kubeadm; please comment here what you think about the planned, separately-invokable phases listed here

luxas · 2017-02-09T19:32:22Z

cc @alexbrand for Kismatic as well :)

luxas · 2017-02-09T19:40:17Z

cc @andrewrynhard

philips · 2017-02-09T23:46:41Z

docs/design/phases.md

+### Generate KubeConfig files on the master
+
+There should be:
+ - a KubeConfig file for kubeadm to use itself and the admin: `/etc/kubernetes/admin.conf`


who is the admin?

It's a kubeconfig file for the cluster operator to copy to any machine where he wants to control the cluster from.

Does kubectl read that file by default for system wide settings? I must have missed that and thought the config had to be in ~/.kube/

philips · 2017-02-09T23:47:08Z

docs/design/phases.md

+### Generate a token.csv file for bootstrapping purposes
+
+There should be:
+ - a `/etc/kubernetes/tokens.csv` file that contains one kubeadm-generated token


gah, how did we end up with a CSV?

I know, I don't like this at all either, but this is what we've got for bootstrapping :/

i'm with @philips on this one; why not json?

@v1k0d3n the apiserver doesn't support reading tokens from json, so that's impossible for the time being

What we all want though, and what we're gonna implement is a dynamic way of managing these tokens so we can ditch this file totally (no one likes handling files on disk)

See https://kubernetes.io/docs/admin/kube-apiserver/

--token-auth-file string If set, the file that will be used to secure the secure port of the API server via token authentication.

The new stuff is here: kubernetes/kubernetes#41281

The usage of this file is gonna be replaced by the kubernetes/kubernetes#36101 and kubernetes/kubernetes#41281

@luxas you should add your last comment to the document.

mattymo · 2017-02-10T14:18:13Z

docs/design/phases.md

+   - contain altnames for
+     - the kubernetes' service's internal clusterIP and dns name (e.g. `10.96.0.1`, `kubernetes.default.svc.cluster.local`, `kubernetes.default.svc`, `kubernetes.default`, `kubernetes`)
+     - the hostname of the node
+     - the IP address of the default gateway


why gateway IP? for snat tricks?

mattymo · 2017-02-10T14:18:32Z

docs/design/phases.md

+     - the kubernetes' service's internal clusterIP and dns name (e.g. `10.96.0.1`, `kubernetes.default.svc.cluster.local`, `kubernetes.default.svc`, `kubernetes.default`, `kubernetes`)
+     - the hostname of the node
+     - the IP address of the default gateway
+     - optional extra altnames that can be specified by the user


Can we include localhost and 127.0.0.1 in this list?

I don't think we want to have a cert signed for localhost. That seems like a security issue.

mattymo · 2017-02-10T14:19:19Z

docs/design/phases.md

+     - the IP address of the default gateway
+     - optional extra altnames that can be specified by the user
+   - be in the `system:masters` organization
+ - a client certificate for the apiservers to connect to the kubelets securely (`apiserver-kubelet-client.crt`) using `ca.crt` as the CA with its private key (`apiserver-kubelet-client.key`). The certificate should:


one cert or one per client?

There's only one client (the apiserver).

justinsb · 2017-02-10T20:38:14Z

docs/design/phases.md

+
+Somewhere we have to draw the line what should be configurable, what shouldn't and what should be hard-coded in the binary.
+
+We've decided to make the Kubernetes directory `/etc/kubernetes` a constant in the application, since it is clearly the given path in a lot of places,


(I'm not even sure you need to justify the choice of /etc/kubernetes)

justinsb · 2017-02-10T20:39:32Z

docs/design/phases.md

+ - an API Server certificate (`apiserver.crt`) using `ca.crt` as the CA with its private key (`apiserver.key`). The certificate should:
+   - be a serving server certificate (`x509.ExtKeyUsageServerAuth`)
+   - contain altnames for
+     - the kubernetes' service's internal clusterIP and dns name (e.g. `10.96.0.1`, `kubernetes.default.svc.cluster.local`, `kubernetes.default.svc`, `kubernetes.default`, `kubernetes`)


Do you mean 100.96.0.1 ? Or is it actually 10.96...?

10.96.0.1/12 is the default service subnet for kubeadm

I know it's the default but do we really need a /12 (10.96.0.0 - 10.111.255.255) for this?

justinsb · 2017-02-10T20:40:39Z

docs/design/phases.md

+   - be a serving server certificate (`x509.ExtKeyUsageServerAuth`)
+   - contain altnames for
+     - the kubernetes' service's internal clusterIP and dns name (e.g. `10.96.0.1`, `kubernetes.default.svc.cluster.local`, `kubernetes.default.svc`, `kubernetes.default`, `kubernetes`)
+     - the hostname of the node


Presumably we aren't changing how kubeadm works here, so this is not the venue, but any notion of "hostname" is a huge red-flag for AWS compatibility

I think this should be optional and not included as a SAN by default.

justinsb · 2017-02-10T20:42:19Z

docs/design/phases.md

+     - the hostname of the node
+     - the IP address of the default gateway
+     - optional extra altnames that can be specified by the user
+   - be in the `system:masters` organization


This precludes/complicates use with an existing CA infrastructure I believe. On the assumption though that doc is about the phases, and this is the existing kubeadm behaviour we are document, I'm going to stop calling out these issues. If that's not right, and we are actually decided what the keys should look like, let me know and I will review with that mindset.

That is an excellent point. We can't be too prescriptive about fields of the certs, since we need to allow certs generated by organizations to work smoothly with kubeadm.

justinsb · 2017-02-10T20:54:51Z

This looks cool. kops could probably use the RBAC generation phase today (although we would want to be sure that it was HA safe, and that it did not reapply if users deleted the RBAC policies). The RBAC layer feels almost like it could be a manifest - perhaps this meshes into the add-on-manager-v2 discussion (where I need to produce some form of strawman design doc based on the existing works).

luxas · 2017-02-10T22:20:40Z

Thanks for the comments everyone, I'll look at this tomorrow or so and update it with your suggestion and more detail 👍

roberthbailey · 2017-02-14T08:07:32Z

docs/design/phases.md

+
+### Certificates
+
+First off, kubeadm generates certificate and private key pairs for different purposes.


Remove the "First off" part of the sentence.

roberthbailey · 2017-02-14T08:09:21Z

docs/design/phases.md

+   - be a serving server certificate (`x509.ExtKeyUsageServerAuth`)
+   - contain altnames for
+     - the kubernetes' service's internal clusterIP and dns name (e.g. `10.96.0.1`, `kubernetes.default.svc.cluster.local`, `kubernetes.default.svc`, `kubernetes.default`, `kubernetes`)
+     - the hostname of the node


I think this should be optional and not included as a SAN by default.

roberthbailey · 2017-02-14T08:09:45Z

docs/design/phases.md

+     - the kubernetes' service's internal clusterIP and dns name (e.g. `10.96.0.1`, `kubernetes.default.svc.cluster.local`, `kubernetes.default.svc`, `kubernetes.default`, `kubernetes`)
+     - the hostname of the node
+     - the IP address of the default gateway
+     - optional extra altnames that can be specified by the user


I don't think we want to have a cert signed for localhost. That seems like a security issue.

roberthbailey · 2017-02-14T08:10:54Z

docs/design/phases.md

+     - the hostname of the node
+     - the IP address of the default gateway
+     - optional extra altnames that can be specified by the user
+   - be in the `system:masters` organization


That is an excellent point. We can't be too prescriptive about fields of the certs, since we need to allow certs generated by organizations to work smoothly with kubeadm.

roberthbailey · 2017-02-14T08:11:08Z

docs/design/phases.md

+     - the IP address of the default gateway
+     - optional extra altnames that can be specified by the user
+   - be in the `system:masters` organization
+ - a client certificate for the apiservers to connect to the kubelets securely (`apiserver-kubelet-client.crt`) using `ca.crt` as the CA with its private key (`apiserver-kubelet-client.key`). The certificate should:


There's only one client (the apiserver).

roberthbailey · 2017-02-14T08:13:02Z

docs/design/phases.md

+   - be a client certificate (`x509.ExtKeyUsageClientAuth`)
+   - be in the `system:masters` organization
+ - a certificate for signing ServiceAccount Tokens (`sa.crt`) using `ca.crt` as the CA with its private key (`sa.key`). The certificate should:
+   - ??


@mikedanese

I think it needs to have the usage for signing certs (KeyUsageCertSign) and shouldn't have any other key usages set.

roberthbailey · 2017-02-14T08:14:20Z

docs/design/phases.md

+
+There should be:
+ - a KubeConfig file for kubeadm to use itself and the admin: `/etc/kubernetes/admin.conf`
+   - inside this file, a client certificate is generated from the `ca.crt` and `ca.key`. The client cert should:


I guess a client cert is ok here for bootstrapping, but once you have the cluster configured you should really throw it away in favor of configuring users so that things like audit logging can track who did what when.

roberthbailey · 2017-02-14T08:15:07Z

docs/design/phases.md

+   - inside this file, a client certificate is generated from the `ca.crt` and `ca.key`. The client cert should:
+     - be a client certificate (`x509.ExtKeyUsageClientAuth`)
+     - be in the `system:masters` organization
+ - a KubeConfig file for kubelet to use: `/etc/kubernetes/kubelet.conf`


If this kubelet is going to connect to the apiserver, shouldn't we use TLS bootstrapping to get it a client cert and give it a temporary token instead?

roberthbailey · 2017-02-14T08:17:06Z

docs/design/phases.md

+ - use an external one if external etcd params are given
+ - spin up one locally if external etcd should not be enabled. It should:
+   - be spun up in a Pod
+   - listen on `localhost:2379`


This will need to change once we support HA. In particular, we'll need to generate another set of certificates (possibly in a different cert universe) for the etcd instances to form a quorum.

luxas · 2017-02-14T20:13:47Z

docs/design/phases.md

+kind: Node
+spec:
+  taints:
+  - key: node-role.kubernetes.io


@davidopp does this sound good to you?

cc @philips @jbeda

What is this taint supposed to signify? If it is for the master node(s) only, then use the same key and value (if any) as you use for the label on the master. (e.g. node-role.kubenetes.io/master with no value)

Also keep in mind that if you add a taint, then all pods that are supposed to run on the node will need a toleration for that taint.

pires

So far this LGTM.

@luxas this is a good job, particularly knowing you're the most active code contributor to kubeadm. Hat off to you!

pires · 2017-02-15T12:25:56Z

docs/design/phases.md

+ - an API Server certificate (`apiserver.crt`) using `ca.crt` as the CA with its private key (`apiserver.key`). The certificate should:
+   - be a serving server certificate (`x509.ExtKeyUsageServerAuth`)
+   - contain altnames for
+     - the kubernetes' service's internal clusterIP and dns name (e.g. `10.96.0.1`, `kubernetes.default.svc.cluster.local`, `kubernetes.default.svc`, `kubernetes.default`, `kubernetes`)


I know it's the default but do we really need a /12 (10.96.0.0 - 10.111.255.255) for this?

pires · 2017-02-15T12:32:55Z

docs/design/phases.md

+   - contain altnames for
+     - the kubernetes' service's internal clusterIP and dns name (e.g. `10.96.0.1`, `kubernetes.default.svc.cluster.local`, `kubernetes.default.svc`, `kubernetes.default`, `kubernetes`)
+     - the hostname of the node
+     - the IPv4 address of the default route


What is this exactly, the apiserver pod IP?

This is what's returned from 0.0.0.0, the IP address from the default (main) net interface

pires · 2017-02-15T12:33:39Z

docs/design/phases.md

+     - the kubernetes' service's internal clusterIP and dns name (e.g. `10.96.0.1`, `kubernetes.default.svc.cluster.local`, `kubernetes.default.svc`, `kubernetes.default`, `kubernetes`)
+     - the hostname of the node
+     - the IPv4 address of the default route
+     - optional extra altnames that can be specified by the user


I haven't seen this in the form of flags or configuration (haven't updated on the configuration API WIP PR). Are we planning ahead?

This has been possible from the beginning by using --api-external-dns-names, but it will probably be exposed in an other shape in the future (via the API Group)

pires · 2017-02-15T12:34:09Z

docs/design/phases.md

+     - the hostname of the node
+     - the IPv4 address of the default route
+     - optional extra altnames that can be specified by the user
+   - ??be in the `system:masters` organization, this might not be needed??


Is DNS integrated with RBAC in any way? I don't think so.

pires · 2017-02-15T12:34:40Z

docs/design/phases.md

+   - ??be in the `system:masters` organization, this might not be needed??
+ - a client certificate for the apiservers to connect to the kubelets securely (`apiserver-kubelet-client.crt`) using `ca.crt` as the CA with its private key (`apiserver-kubelet-client.key`). The certificate should:
+   - be a client certificate (`x509.ExtKeyUsageClientAuth`)
+   - be in the `system:masters` organization


Is DNS integrated with RBAC in any way? I don't think so.

This isn't related to DNS, but RBAC. Being in the system:master Organization (maps to Group in RBAC), basically makes you use sudo while talking to the cluster.

pires · 2017-02-15T12:40:02Z

docs/design/phases.md

+ - spin up one locally if external etcd is not be enabled. It should:
+   - be spun up in a Pod
+   - listen on `localhost:2379`
+   - set `PodSpec.SecurityContext.SELinuxOptions.Type=spc_t`


Can you provide a link or a little explanation why?

pires · 2017-02-15T12:41:18Z

docs/design/phases.md

+```
+
+If self-hosting should be enabled, do a little bit more:
+ - TODO


I should help here. When does this doc need to be merged?

When the design is fully settled and agreed on :)

pires · 2017-02-15T12:41:50Z

docs/design/phases.md

+
+### Configure the bootstrap tokens
+
+This phase configures the BootstrapSigner to create the public `cluster-info` ConfigMap in the `kube-public` namespace


Can you please provide a link, even if to just the PR, to what the BootstrapSigner is?

pires · 2017-02-15T12:42:42Z

docs/design/phases.md

+In the future, the base layer of configuration for all kubelets in the cluster will be set in this phase.
+
+Then the rest of the addons follow:
+ - apply the CNI Network Provider file if specified


Please add, make sure CNI plug-in supports the new RBAC mode if RBAC is enabled.

pires · 2017-02-15T12:43:14Z

docs/design/phases.md

+ - apply the CNI Network Provider file if specified
+ - deploy kube-proxy as a DaemonSet
+   - the credentials to the master come from the ServiceAccount
+   - the location of the master comes from a ConfigMap


We also need the pod network or is this gone now?

deads2k · 2017-02-15T20:07:28Z

docs/design/phases.md

+First off, kubeadm generates certificate and private key pairs for different purposes.
+Certificates are stored by default in `/etc/kubernetes/pki`.
+
+There should be:


There should also be a CA keypair and client certificate for the Authenticating Proxy described here: https://kubernetes.io/docs/admin/authentication/#authenticating-proxy. It's near zero cost in the kube-apiserver to have it, there's no exposure unless actual machine access is had, and future extension stories (https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/kubernetes-sig-api-machinery/o2v-HjAHk0k/Rg4lNeEACQAJ) rely upon it.

davidopp · 2017-02-17T09:36:03Z

docs/design/phases.md

+
+The master node is patched, the master label is added, and the master is tainted.
+
+The proposed master label for `v1.6` is `node-role.kubernetes.io/master=true`.


Having a value of "true" is probably not useful. Why not just have the key (node-role.kubernetes.io/master) and no value?

It makes so much sense to me that I now feel dumb because I didn't think about it first.

luxas · 2017-02-20T16:59:31Z

Updated this now with a lot more detail about what should be done in kubeadm.
Also includes proposal for kubeadm phase

jbeda · 2017-02-20T19:44:06Z

docs/design/phases.md

+ - `--admission-control` to `NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota` or whatever the recommended set of admission controllers is at the time
+ - `--storage-backend` to `etcd3`. Support for `etcd2` in kubeadm is dropped.
+ - `--kubelet-preferred-address-types` to `InternalIP,ExternalIP,Hostname` so `kubectl logs` and other apiserver -> kubelet communication works in environments where the hostnames of the nodes aren't resolvable.
+ - `--requestheader-username-headers=X-Remote-User`, `--requestheader-group-headers=X-Remote-Group`, `--requestheader-extra-headers-prefix=X-Remote-Extra-`, --requestheader-allowed-names=front-proxy-client` so the front proxy communication is secure.


Missing a "`" here.

jbeda

This looks pretty darn good to me.

One question on UX and please correct me if I'm wrong. Here are my assumptions:

There will be a limited number of flags for kubeadm init and kubeadm join. This is to keep this surface area simple.
There will be extra options exposed as flags on individual phases.
If users want access to those options they have 2 choices:
- Run the phases individually
- Create a config file (different ones for join and init) and pass that to the top level command.

jbeda · 2017-02-20T22:54:07Z

docs/design/phases.md

+   - `--use-kubernetes-version=${STABLE_VERSION}`
+   - `--self-hosted`
+
+What would `kubeadm init` actually do?


Not 100% necessary for v1 of this but it would be awesome if there were a kubeadm init --plan that would just output this for the options that are passed in. In this case users can understand what kubeadm is planning on doing in terms of the phases and make it clear that kubeadm is really just a bundle command.

timothysc · 2017-02-22T19:18:03Z

docs/design/phases.md

+     - the hostname of the node
+       - I guess this might be a requested feature in opinionated setups, but might be a no-no in more advanced setups. Consensus here?
+     - the IPv4 address of the default route
+     - optional extra altnames that can be specified by the user


should describe how. Ideally not just a massive command line arg list, but also a config file.

Everything described here will be available to config via the configfile which is going to expose a nice (the nicest) way of actually setting a kubeadm cluster up.
I should point that out much better...

timothysc · 2017-02-22T19:36:28Z

docs/design/phases.md

+ - listen on `localhost:2379` and use `HostNetwork=true`
+ - have `PodSpec.SecurityContext.SELinuxOptions.Type=spc_t` set because of https://github.com/kubernetes/kubeadm/issues/107
+ - be at a minimum version `3.0.14`
+ - make a volume out to the host on a given host path


We need to be certain we don't resource starve CPU on etcd + we need to make certain we open up ulimit caps ~ 65536 as a sane default.

@timothysc Could you send a quick PR on the ulimit thing for etcd?

So I dug on this and found out that we still don't support ulimit on pods, but we can override dockers startup defaults.. :-/

fabriziopandini · 2017-04-13T21:29:00Z

docs/design/phases.md

+   - `--use-kubernetes-version=${STABLE_VERSION}`
+   - `--self-hosted`
+
+What would `kubeadm init` actually do?


Nice to have: Additional aliases for each phase with a numbered prefix. e.g.

kubeadm phase 1-certs ... (where 1-certs is alias for certscertificates`)

kubeadm phase 2-kubeconfig ... (where 2-kubeconfig is alias for kubeconfig)

Those aliases could help users in understanding/using the right sequence of phases, and also in having a better experience with autocompletion.

fabriziopandini · 2017-04-13T22:33:58Z

docs/design/phases.md

+     - `--image-repository=gcr.io/google_containers`
+   - Outputs:
+     - Applies the built-in `kube-dns` manifest to the API Server
+


I suggest to provide a more consistent approach for outputs across the entire kubeadm phase UX, harmonising current approaches:

kubeadm phase certs selfsign use --cert-dir=

kubeadm phase kubeconfig clientcert writes to stdout

kubeadm phase controlplane * writes to /etc/kubernetes/manifests/*

kubeadm phase addons cluster-info apply directly to the apiserver

eg

A first draft for possible alternative solution is:

by default all kubeadm phase commands should provide the same experience of kubeadm init:

kubeadm phase certs and kubeadm phase token writes to --cert-dir, which defaults to /etc/kubernetes/pki

kubeadm phase kubeconfig clientcert writes to /etc/kubernetes/; to be defined a rule for deriving filename from --client-name=(this is easier to implement if we will specialize kubeconfig phase)

kubeadm phase controlplane * writes to /etc/kubernetes/manifests/*.yaml

kubeadm phase addons * apply directly to the apiserver

if invoked with a --stdoutflag, all kubeadm phase commands should write to stdout; the output for the most of the commands is a yaml string or a sequence of kubectl commands; to be defined a format for kubeadm phase certs (alternative support --stdout only where it make sense)

fabriziopandini · 2017-04-13T22:41:28Z

docs/design/phases.md

+   - Outputs:
+     - Creates the `cluster-info` ConfigMap in the `kube-public` namespace
+     - Exposes it publicly with RBAC bindings
+- `kubeadm phase addons self-hosted`


might be that considering self-hosted as an addons can be confusing for users.
I suggest to treat self-hosted phase separately from addons

fabriziopandini · 2017-04-13T23:02:39Z

docs/design/phases.md

+     - `sa.{pub,key}`
+     - `front-proxy-ca.{crt,key}`
+     - `front-proxy-client.{crt,key}`
+ - `kubeadm phase kubeconfig clientcert`


Not sure if offering a generic kubeconfig clientcert/token phase can offer the simplest choice for users, because it requires to call this phase 4 times (kubelet,scheduler,controller-manager,admin), each one with very specific parameters.

A draft alternative proposal, might be specializing kubeconfig phase with dedicate sub-commands:

kubeconfig kubelet (Inputs: --api-server, --cert-dir)

kubeconfig scheduler (Inputs: --api-server, --cert-dir)

kubeconfig controller-manager (Inputs: --api-server, --cert-dir)

kubeconfig admin (Inputs: --api-server, --cert-dir)
All commands should produce a clientCert (default mode) expet if if --token flag is provided)

Additionally

it could be useful to support a kubeconfig all (or simply kubeconfig without subcommands) allowing to generate all 4 default config files in a single action

it could be useful to support a kubeconfig user (Inputs: --api-server, --cert-dir, --name, --organization), allowing to create kubeconfig files fore other admins/other cluster users

for kubeconfig admin and kubeconfig user it could be useful to to generate also an alternative output format, providing kubectl config set-cluster, kubectl config set credentials commands linking to separated certificates files (e.g. useful for admin managing more that one cluster... > with a kubeconfig containing many clusters/credentitals)

fabriziopandini · 2017-04-13T23:03:32Z

docs/design/phases.md

+     - `--cert-dir=/etc/kubernetes/pki`
+   - Output:
+     - The kubeconfig is `stdout`
+ - `kubeadm phase controlplane etcd`


it could be useful to support a controlplane all (or simply controlplane without subcommands) allowing to generate all control plane components in a single action

fabriziopandini · 2017-04-14T08:40:11Z

docs/design/phases.md

+     - The kubeconfig is `stdout`
+ - `kubeadm phase controlplane etcd`
+   - Inputs:
+     - `--image=gcr.io/google_containers/etcd-${GOARCH}:3.0.14-kubeadm`


I suggest to keep an approach more consistent with kubeadm init, providing flags --image-repository=gcr.io/google_containersand --use-kubernetes-version=${STABLE_VERSION}instead of--image=...`

fabriziopandini · 2017-04-14T09:07:51Z

docs/design/phases.md

+ - `kubeadm phase controlplane etcd`
+   - Inputs:
+     - `--image=gcr.io/google_containers/etcd-${GOARCH}:3.0.14-kubeadm`
+     - `--data-dir=/var/lib/etcd`


By extending etcd flags, it is possible to provide with a minimal effort an early support for creating HA clusters, e.g.:

With kubeadmin init, create the first master with

--apiserver-advertise-address and --cert-altnames properly set

an etcd.yaml that contains instructions for spinning up an etcd instance that should be member of an etcd cluster

Manually copy /etcd/kubernetes folder to a second and third masters (probably also necessary to replace kubelet.conf with a new one containing new credentials, created with kubeconfig user)

Manually setup a LB/DNS in front of all api-servers (matching with --apiserver-advertise-address and --cert-altnames)

Join nodes

In order to support this a possibile solution is:

add --name flag to etcd command, and assign to it a unique name (e.g. the nodename value retrived from Downward API)

add a --etcd-cluster-discovery flag supporting a dynamic approach for setting up an etcd cluster (no need of knowing the name/ip of nodes in advance, but access to internet/to another etcd instance required)

add a `--etcd-cluster-members' flag supporting a static approach for setting up an etcd cluster (required knowing the name/ip of nodes in advance, no need of access to internet/to another etcd instance)

add other flag to etcd command required in a cluster configuration (--initial-advertise-peer-urls and
--listen-peer-urls)

fabriziopandini · 2017-04-14T09:10:18Z

docs/design/phases.md

+	--image=${IMAGE_REPOSITORY}/kube-scheduler-${ARCH}:${USE_KUBERNETES_VERSION}
+
+kubectl taint node $(hostname -i) node-role.kubernetes.io/master:NoSchedule
+kubectl label node $(hostname -i) node-role.kubernetes.io/master=""


Using kubectlin the middle of a kubeadmsequence of phasessounds a little bit strange. With low priority (but also low effort), it might make sense to implement a phase also for this step.

fabriziopandini · 2017-04-14T10:06:57Z

docs/design/phases.md

+This phase creates the `cluster-info` ConfigMap in the `kube-public` namespace as defined in https://github.com/kubernetes/community/blob/master/contributors/design-proposals/bootstrap-discovery.md
+ - The `ca.crt` and the address/port of the apiserver is added to the `cluster-info` ConfigMap in the `kubeconfig` key
+ - Exposes the `cluster-info` ConfigMap to unauthenticated users (i.e. users in RBAC group `system:unauthenticated`)
+


Looking at kubeadm init code, it seems there are also two additional steps here: CreateServiceAccounts and CreateRBACRules.
In order to make kubeadm init a full "bundle command", also those steps should be implemented as phases.

timothysc · 2017-06-13T15:37:51Z

I'm of the opinion we should merge this and move along with updates as 1.8 progresses.

mtaufen · 2017-06-13T15:58:58Z

docs/design/design.md

+`kubeadm init` and `kubeadm join` together provides a nice user experience for creating a best-practice but bare Kubernetes cluster from scratch.
+However, it might not be obvious _how_ kubeadm does that.
+
+This document strives to explain what types of phases of work happen under the hood.


suggest: "This document strives to explain the phases of work that happen under the hood."

mtaufen · 2017-06-13T16:01:17Z

docs/design/design.md

+However, it might not be obvious _how_ kubeadm does that.
+
+This document strives to explain what types of phases of work happen under the hood.
+Also included is ComponentConfiguration API types for talking to kubeadm programmatically.


Are you relying on the componentconfig API group? Be careful with this, that group will be deleted soon (as soon as every component's config has been moved out of it).

No, it's what I think the new ComponentConfiguration stuff looks like: https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/v1alpha1/types.go

in it's own API group and everything. Not coupled to anything else, just reusing the API machinery

Ah, ok nice!

mtaufen · 2017-06-13T16:11:55Z

docs/design/design.md

+     - The `kubeadm join` request to add a node should be automatically approved
+ - Extendable
+   - It should for example _not_ favor any network provider, instead configuring a network is out-of-scope
+   - Should provide a config file that can be used for customizing various parameters


Is this versioned?

yes, see the code above

mtaufen · 2017-06-13T16:42:22Z

This is really cool!

luxas · 2017-06-13T18:08:11Z

Thank you @mtaufen!

@timothysc Totally agree -- will merge now and follow up as we go.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 9, 2017

dmmcquay reviewed Feb 9, 2017

View reviewed changes

philips reviewed Feb 9, 2017

View reviewed changes

mattymo reviewed Feb 10, 2017

View reviewed changes

justinsb reviewed Feb 10, 2017

View reviewed changes

v1k0d3n mentioned this pull request Feb 11, 2017

feat: changes in kubeadm are coming att-comdev/halcyon-vagrant-kubernetes#58

Open

roberthbailey reviewed Feb 14, 2017

View reviewed changes

luxas commented Feb 14, 2017

View reviewed changes

pires suggested changes Feb 15, 2017

View reviewed changes

deads2k reviewed Feb 15, 2017

View reviewed changes

davidopp reviewed Feb 17, 2017

View reviewed changes

This was referenced Feb 20, 2017

kubeadm: Remove the --cloud-provider flag for beta init UX kubernetes/kubernetes#41710

Merged

kubeadm: Add a --ca-cert-path flag to kubeadm join kubernetes/kubernetes#41721

Merged

jbeda reviewed Feb 20, 2017

View reviewed changes

jbeda approved these changes Feb 20, 2017

View reviewed changes

jbeda reviewed Feb 20, 2017

View reviewed changes

pires approved these changes Feb 21, 2017

View reviewed changes

timothysc reviewed Feb 22, 2017

View reviewed changes

fabriziopandini reviewed Apr 13, 2017

View reviewed changes

fabriziopandini reviewed Apr 14, 2017

View reviewed changes

luxas mentioned this pull request Apr 30, 2017

kubeadm should allow using custom certificates #239

Closed

lewismarshall mentioned this pull request Jun 12, 2017

Changes required upstream to move to kubeadm as a CLI not a lib nonlive/keto-k8#31

Open

4 tasks

luxas changed the title ~~Add a doc about the different phases in kubeadm~~ Add a doc about the kubeadm design and phases implementation Jun 13, 2017

luxas mentioned this pull request Jun 13, 2017

kubeadm: Move directory ./master to ./phases/controlplane kubernetes/kubernetes#47345

Merged

mtaufen reviewed Jun 13, 2017

View reviewed changes

Add a design doc for how kubeadm works

4ec0ef9

luxas force-pushed the add_phases branch from 8d06322 to 4ec0ef9 Compare June 13, 2017 18:07

luxas merged commit 34abbf7 into kubernetes:master Jun 13, 2017

This was referenced Jul 20, 2018

Authorization error with multiple masters #1013

Closed

Offline install with kubeadm v1.11 fails #1019

Closed

sudeeshjohn mentioned this pull request Jul 27, 2018

kubeadm init --kubernetes-version ci-cross/latest fails to pull the images #1024

Closed

This was referenced Aug 2, 2018

unable to generate certificates offline #1032

Closed

apiserver-etcd-client certificates differ #1033

Closed

		@@ -0,0 +1,110 @@
		## Phases in kubeadm

		`kubeadm init` does a lot of things. In order to make kubeadm the common building block for Kubernetes we want it to be, each logical phase kubeadm does during `init` should be invokable separately. Here's an explanation what kubeadm does in the different phases:


		#### A note on constants / well-known values and paths

		Somewhere we have to draw the line what should be configurable, what shouldn't and what should be hard-coded in the binary.


		Somewhere we have to draw the line what should be configurable, what shouldn't and what should be hard-coded in the binary.

		We've decided to make the Kubernetes directory `/etc/kubernetes` a constant in the application, since it is clearly the given path in a lot of places,


		### Bootstrap the control plane

		First, see so there is an etcd cluster somewhere:


		Then kubeadm waits for the control plane to come up as static pods

		The master node is patched; the master label is added and the master is tainted.


		### Certificates

		First off, kubeadm generates certificate and private key pairs for different purposes.


		### Configure the bootstrap tokens

		This phase configures the BootstrapSigner to create the public `cluster-info` ConfigMap in the `kube-public` namespace


		The master node is patched, the master label is added, and the master is tainted.

		The proposed master label for `v1.6` is `node-role.kubernetes.io/master=true`.

Add a doc about the kubeadm design and phases implementation #156

Add a doc about the kubeadm design and phases implementation #156

Conversation

luxas commented Feb 9, 2017

dmmcquay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luxas commented Feb 9, 2017

luxas commented Feb 9, 2017

luxas commented Feb 9, 2017

Choose a reason for hiding this comment

pires Feb 10, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justinsb commented Feb 10, 2017

luxas commented Feb 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pires left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pires Feb 10, 2017 •

edited

Loading