-
Notifications
You must be signed in to change notification settings - Fork 5.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update bootstrap design doc with kubeadm UX #381
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -18,13 +18,13 @@ Similarly, mature organizations will be able to rely on a centrally managed DNS | |
|
||
With that in mind, the proposals here will devolve into simply using DNS names that are validated with system installed root certificates. | ||
|
||
## Cluster Location information | ||
## Cluster location information (aka ClusterInfo) | ||
|
||
First we define a set of information that identifies a cluster and how to talk to it. | ||
First we define a set of information that identifies a cluster and how to talk to it. We will call this ClusterInfo in this document. | ||
|
||
While we could define a new format for communicating the set of information needed here, we'll start by using the standard [`kubeconfig`](http://kubernetes.io/docs/user-guide/kubeconfig-file/) file format. | ||
|
||
It is expected that the `kubeconfig` file will have a single unnamed `Cluster` entry. Other information (especially authentication secrets) must be omitted. | ||
It is expected that the `kubeconfig` file will have a single unnamed `Cluster` entry. Other information (especially authentication secrets) MUST be omitted. | ||
|
||
### Evolving kubeconfig | ||
|
||
|
@@ -45,7 +45,7 @@ Additions include: | |
|
||
**This is to be implemented in a later phase** | ||
|
||
Any client of the cluster will want to have this information. As the configuration of the cluster changes we need the client to keep this information up to date. It is assumed that the information here won't drift so fast that clients won't be able to find *some* way to connect. | ||
Any client of the cluster will want to have this information. As the configuration of the cluster changes we need the client to keep this information up to date. The ClusterInfo ConfigMap (defined below) is expected to be a common place to get the latest ClusterInfo for any cluster. Clients should periodically grab this and cache it. It is assumed that the information here won't drift so fast that clients won't be able to find *some* way to connect. | ||
|
||
In exceptional circumstances it is possible that this information may be out of date and a client would be unable to connect to a cluster. Consider the case where a user has kubectl set up and working well and then doesn't run kubectl for quite a while. It is possible that over this time (a) the set of servers will have migrated so that all endpoints are now invalid or (b) the root certificates will have rotated so that the user can no longer trust any endpoint. | ||
|
||
|
@@ -55,31 +55,35 @@ Now that we know *what* we want to get to the client, the question is how. We w | |
|
||
### Method: Out of Band | ||
|
||
The simplest way to do this would be to simply put this object in a file and copy it around. This is more overhead for the user, but it is easy to implement and lets users rely on existing systems to distribute configuration. | ||
The simplest way to obtain ClusterInfo this would be to simply put this object in a file and copy it around. This is more overhead for the user, but it is easy to implement and lets users rely on existing systems to distribute configuration. | ||
|
||
For the `kubeadm` flow, the command line might look like: | ||
|
||
``` | ||
kubeadm join --cluster-info-file=my-cluster.yaml | ||
kubeadm join --discovery-file=my-cluster.yaml | ||
``` | ||
|
||
Note that TLS bootstrap (which establishes a way for a client to authenticate itself to the server) is a separate issue and has its own set of methods. This command line may have a TLS bootstrap token (or config file) on the command line also. | ||
After loading the ClusterInfo from a file, the client MAY look for updated information from the server by reading the `kube-public` `cluster-info` ConfigMap defined below. However, when retrieving this ConfigMap the client MUST validate the certificate chain when talking to the API server. | ||
|
||
**Note:** TLS bootstrap (which establishes a way for a client to authenticate itself to the server) is a separate issue and has its own set of methods. This command line may have a TLS bootstrap token (or config file) on the command line also. For this reason, even thought the `--discovery-file` argument is in the form of a `kubeconfig`, it MUST NOT contain client credentials as defined above. | ||
|
||
### Method: HTTPS Endpoint | ||
|
||
If the ClusterInfo information is hosted in a trusted place via HTTPS you can just request it that way. This will use the root certificates that are installed on the system. It may or may not be appropriate based on the user's constraints. | ||
If the ClusterInfo information is hosted in a trusted place via HTTPS you can just request it that way. This will use the root certificates that are installed on the system. It may or may not be appropriate based on the user's constraints. This method MUST use HTTPS. Also, even though the payload for this URL is the `kubeconfig` format, it MUST NOT contain client credentials. | ||
|
||
``` | ||
kubeadm join --cluster-info-url="https://example/mycluster.yaml" | ||
kubeadm join --discovery-url="https://example/mycluster.yaml" | ||
``` | ||
|
||
This is really a shorthand for someone doing something like (assuming we support stdin with `-`): | ||
|
||
``` | ||
curl https://example.com/mycluster.json | kubeadm join --cluster-info-file=- | ||
curl https://example.com/mycluster.json | kubeadm join --discovery-file=- | ||
``` | ||
|
||
If the user requires some auth to the HTTPS server (to keep the ClusterInfo object private) that can be done in the curl command equivalent. Or we could eventually add it to `kubeadm` directly. | ||
After loading the ClusterInfo from a URL, the client MAY look for updated information from the server by reading the `kube-public` `cluster-info` ConfigMap defined below. However, when retrieving this ConfigMap the client MUST validate the certificate chain when talking to the API server. | ||
|
||
**Note:** support for loading from stdin for `--discovery-file` may not be implemented immediately. | ||
|
||
### Method: Bootstrap Token | ||
|
||
|
@@ -100,7 +104,7 @@ The user experience for joining a cluster would be something like: | |
kubeadm join --token=ae23dc.faddc87f5a5ab458 <address> | ||
``` | ||
|
||
**Note:** This is logically a different use of the token from TLS bootstrap. We harmonize these usages and allow the same token to play double duty. | ||
**Note:** This is logically a different use of the token used for authentication for TLS bootstrap. We harmonize these usages and allow the same token to play double duty. | ||
|
||
#### Implementation Flow | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (Commenting here as github won't let me comment on the other lines)
This is not a good idea. Unauthenticated access to the same endpoint always evolves into a security vulnerability. I presume this is not enabled by default, or that it can be turned off? Have the k8s security folk signed off on this? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes -- approved and in the merge queue. See kubernetes/kubernetes#41281 |
||
|
@@ -130,6 +134,8 @@ The first part of the token is the `token-id`. The second part is the `token-se | |
|
||
This new type of token is different from the current CSV token authenticator that is currently part of Kubernetes. The CSV token authenticator requires an update on disk and a restart of the API server to update/delete tokens. As we prove out this token mechanism we may wish to deprecate and eventually remove that mechanism. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree that we should make tokens dynamic. But I am unclear why bootstrap tokens are different from "normal" auth tokens? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Normal auth tokens are defined in a CSV file on disk. We are scoping this mechanism to just bootstrap as we are storing the tokens in etcd. The general recommended approach is, for day to day usage, use TLS client certs or another more full featured auth provider. |
||
|
||
The `token-id` must be 6 characters and the `token-secret` must be 16 characters. They must be lower case ASCII letters and numbers. Specifically it must match the regular expression: `[a-z0-9]{6}\.[a-z0-9]{16}`. There is no strong reasoning behind this beyond the history of how this has been implemented in alpha versions. | ||
|
||
#### NEW: Bootstrap Token Secrets | ||
|
||
Bootstrap tokens are stored and managed via Kubernetes secrets in the `kube-system` namespace. They have type `bootstrap.kubernetes.io/token`. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is the plan to move this to an API type? Can we do it now? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I wanted to start as an API type from the start but after a lot of discussion over many months we settled on this. You can read the super long gory history here: kubernetes/kubernetes#30707 |
||
|
@@ -138,11 +144,13 @@ The following keys are on the secret data: | |
* **token-id**. As defined above. | ||
* **token-secret**. As defined above. | ||
* **expiration**. After this time the token should be automatically deleted. This is encoded as an absolute UTC time using RFC3339. | ||
* **usage-bootstrap-signing**. Set to `true` to indicate this token should be used for signing bootstrap configs. If omitted or some other string, it defaults to `false`. | ||
* **usage-bootstrap-signing**. Set to `true` to indicate this token should be used for signing bootstrap configs. If this is missing from the token secret or set to any other value, the usage is not allowed. | ||
* **usage-bootstrap-authentication**. Set to true to indicate that this token should be used for authenticating to the API server. If this is missing from the token secret or set to any other value, the usage is not allowed. The bootstrap token authenticagtor will use this token to auth as a user that is `system:bootstrap:<token-id>` in the group `system:bootstrappers`. | ||
* **description**. An optional free form description field for denoting the purpose of the token. If users have especially complex token management neads, they are encouraged to use labels and annotations instead of packing machined readable data in to this field. | ||
|
||
These secrets can be named anything but it is suggested that they be named `bootstrap-token-<token-id>`. | ||
**Future**: At some point in the future we may add the ability to specify a set of groups that this token part of during authentication. This will allow users to segment off which tokens are allowed to bootstrap which nodes. However, we will restrict these groups under `system:bootstrappers:*` to discourage usage outside of bootstrapping. | ||
|
||
**QUESTION:** Should we also spec out now how we can use this token for TLS bootstrap. | ||
These secrets MUST be named `bootstrap-token-<token-id>`. If a token doesn't adhere to this naming scheme it MUST be ignored. The secret MUST also be ignored if the `token-id` key in the secret doesn't match the name of the secret. | ||
|
||
#### Quick Primer on JWS | ||
|
||
|
@@ -167,11 +175,60 @@ A new well known ConfigMap will be created in the `kube-public` namespace called | |
|
||
Users configuring the cluster (and eventually the cluster itself) will update the `kubeconfig` key here with the limited `kubeconfig` above. | ||
|
||
A new controller is introduced that will watch for both new/modified bootstrap tokens and changes to the `cluster-info` ConfigMap. As things change it will generate new JWS signatures. These will be saved under ConfigMap keys of the pattern `jws-kubeconfig-<token-id>`. | ||
A new controller (`bootstrapsigner`) is introduced that will watch for both new/modified bootstrap tokens and changes to the `cluster-info` ConfigMap. As things change it will generate new JWS signatures. These will be saved under ConfigMap keys of the pattern `jws-kubeconfig-<token-id>`. | ||
|
||
Another controller (`tokencleaner`) is introduced that deletes tokens that are past their expiration time. | ||
|
||
Logically these controllers could run as a component in the control plane. But, for the sake of efficiency, they are bundeled as part of the Kubernetes controller-manager. | ||
|
||
## `kubeadm` UX | ||
|
||
We extend kubeadm with a set of flags and helper commands for managing and using these tokens. | ||
|
||
### `kubeadm init` flags | ||
|
||
* `--token` If set, this injects the bootstrap token to use when initializing the cluster. If this is unset, then a random token is created and shown to the user. If set explicitly to the empty string then no token is generated or created. This token is used for both discovery and TLS bootstrap by having `usage-bootstrap-signing` and `usage-bootstrap-authentication` set on the token secret. | ||
* `--token-ttl` If set, this sets the TTL for the lifetime of this token. Defaults to 0 which means "forever" | ||
|
||
### `kubeadm join` flags | ||
|
||
* `--token` This sets the token for both discovery and bootstrap auth. | ||
* `--discovery-url` If set this will grab the cluster-info data (a kubeconfig) from a URL. Due to the sensitive nature of this data, we will only support https URLs. This also supports `username:password@host` syntax for doing HTTP auth. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmm, can we rename these to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is discovery, not tls bootstrap. Discussion below. |
||
* `--discovery-file` If set, this will load the cluster-info from a file. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same goes for this arg: can it be |
||
* `--discovery-token` If set, (or set via `--token`) then we will be using the token scheme described above. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry if I was unclear last review (I was in a hurry), but I do think we should keep what you had in terms of exposing both This way, the only
I propose to define this in the doc and refer to 1. as So TL;DR; There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm confused here. Discovery == "getting and trusting the cluster-info file". We have 3 ways to do this: token, HTTPS endpoint (trusting the system CA bundle) and local file. This is driven by the I think you are wrong wrt 2. The TLS bootstrap dance only works with a token. It is what is used to authenticate to the API server using temporary credentials (the token) in order to submit a CSR. That would be the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok, I see how you're thinking. DiscoveryScenario A: We only have a token and an address to an API Server. What do we do? We construct a KubeConfig file like this: apiVersion: v1
kind: Config
clusters:
- cluster:
server: https://BOOTSTRAP_SERVER:PORT
insecureTLSVerify: false
name: discovery
contexts:
- context:
cluster: discovery
user: ""
name: discovery
current-context: discovery and do
in order to get the KubeConfig with the CA cert and cluster addresses + the JWS token. If everything succeeds, we'll end up with a KubeConfig file like this: apiVersion: v1
kind: Config
clusters:
- cluster:
server: https://SERVER:PORT{https://SERVER2:PORT2,...}
certificate-authority-data: <cacertbytes>
name: tlsbootstrap-ready
users:
- name: tlsbootstrap-ready
user:
token: <tls-bootstrap-token-id>:<tls-bootstrap-token-secret>
contexts:
- context:
cluster: tlsbootstrap-ready
user: tlsbootstrap-ready
name: tlsbootstrap-ready
current-context: tlsbootstrap-ready This is what we need for the next phase (TLS Bootstrap). I.e. this is the output of the discovery phase and the input to the TLS bootstrap phase in my world. Scenario B: We only have a KubeConfig file with credentials that we can use to talk in a secure manner to the cluster. What do we do? Well, since we have all the information needed for TLS Bootstrap already, we'll just skip this phase and go on to the next. At least, this is what we do now in kubeadm. I don't know, maybe it makes sense to hit the cluster-info ConfigMap regardless in order to check for updated information (which can be more than the CA and the apiserver list in the future). If that was what you thought, then I support that. But FYI, that's far from what's implemented now. TLS BootstrapThis is just "the normal" TLS bootstrap flow, which in the future may/should be implemented by the kubelet itself for easier lifecycle management of those certs. The only thing we need as an input here is a KubeConfig file that has:
If we should hit the cluster-info endpoint regardless of Anyway I think it's very important to define the inputs/outputs here. WDYT about this? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Wrt Discovery -- This is all outlined in the first section of the document. This is about distributing information about the cluster itself including the CA bundle for that server. Over time we can use this to find a set of API servers but that is for the future. This has nothing to do with auth at all and it is bad news if you put TLS client certificates in cluster-info. Earlier in the doc I say "It is expected that the But, once you find a cluster, how do you get a unique client certificate signed by a CA that the server trusts? That is the TLS bootstrap flow. And the option we are using here os to also use the token for that. If you have a client certificate that you've already gotten via other means we should support that. But that really isn't "bootstrap" in any way. The client TLS stuff is a bit out of scope for this doc but I'd be cool with a set of flags like this:
If either There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Sorry, that was not what I meant. I meant that you could make a TLS-Bootstrap-only client cert (instead of a token) in your KubeConfig file that will be used for TLS bootstrapping i.e. in the second phase of what kubeadm does. I just said that I think this is possible as an answer to:
I agree those flags would be really cool. |
||
* `--tls-bootstrap-token` (not officially part of this spec) This sets the token used to temporarily authenticate to the API server in order to submit a CSR for signing. If `--insecure-experimental-approve-all-kubelet-csrs-for-group` is set to `system:bootstrappers` then these CSRs will be approved automatically for a hands off joining flow. | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When specifying some kind of token, one also have to specify where the master is located. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Github glitch made me miss this before I merged. Needed to reload page. But I think we talked about this on slack and decided to keep the CLI flags consistent with current behavior where the master to contact with the token isn't behind a flag. |
||
Only one of `--discovery-url`, `--discovery-file` or `--discovery-token` can be set. If more than one is set then an error is surfaced and `kubeadm join` exits. Setting `--token` counts as setting `--discovery-token`. | ||
|
||
### `kubeadm token` commands | ||
|
||
`kubeadm` provides a set of utilities for manipulating token secrets in a running server. | ||
|
||
* `kubeadm token create [token]` Creates a token server side. With no options this'll create a token that is used for discovery and TLS bootstrap. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't want to bloat RBAC into this spec either, but would it make sense to add a
that would be applied in the same go if this arg is specified. This way the user could easily attach an extra role... Well, feel free to ignore this, it was just a crazy idea that sprung to mind :) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think we want to encourage people to use these tokens outside of just bootstrapping. We can always add it later if we find another bootstrap-ish use for it, but, for now, I'd like to keep these scoped. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm ok with that |
||
* `[token]` The actual token value (in `id.secret` form) to write in. If unset, a random value is generated. | ||
* `--usages` A list of usages. Defaults to `signing,authentication`. | ||
* If the `signing` usage is specified, the token will be used (by the BootstrapSigner controller in the KCM) to JWS-sign the ConfigMap and can then be used for discovery. | ||
* If the `authentication` usage is specified, the token can be used to authenticate for TLS bootstrap. | ||
* `--ttl` The TTL for this token. This sets the expiration of the token as a duration from the current time. This is converted into an absolute UTC time as it is written into the token secret. | ||
* `--description` Sets the free form description field for the token. | ||
* `kubeadm token delete <token-id>|<token-id>.<token-secret>` | ||
* Users can either just specify the id or the full token. This will delete the token if it exists. | ||
* `kubeadm token list` | ||
* List tokens in a table form listing out the `token-id.token-secret`, the TTL, the absolute expiration time, the usages, and the description. | ||
* **Question** Support a `--json` or `-o json` way to make this info programmatic? We don't want to recreate `kubectl` here and these aren't plain API objects so we can't reuse that plumbing easily. | ||
* `kubeadm token generate` This currently exists but is documented here for completeness. This pure client side method just generated a random token in the correct form. | ||
|
||
## Implementation Details | ||
|
||
Our documentations (and output from `kubeadm`) should stress to users that when the token is configured for authenitication and used for TLS bootstrap (using `--insecure-experimental-approve-all-kubelet-csrs-for-group`) it is essentially a root password on the cluster and should be protected as such. Users should set a TTL to limit this risk. Or, after the cluster is up and running, users should delete the token using `kubeadm token delete`. | ||
|
||
After some back and forth, we decided to keep the separator in the token between the ID and Secret be a `.`. During the 1.6 cycle, at one point `:` was implemented but then reverted. | ||
|
||
See https://github.com/kubernetes/client-go/issues/114 for details on creating a shared package with common constants for this scheme. | ||
|
||
In addition, `jws-kubeconfig-<token-id>-hash` will be set to the MD5 hash of the contents of the `kubeconfig` data. This will be in the form of `md5:d3b07384d113edec49eaa6238ad5ff00`. This is done so that the controller can detect which signatures need to be updated without reading all of the tokens. | ||
This proposal assumes RBAC to lock things down in a couple of ways. First, it will open up `cluster-info` ConfigMap in `kube-public` so that it is readable by unauthenticated users. Next, it will make it so that the identities in the `system:bootstrappers` group can only be used with the certs API to submit CSRs. After a TLS certificate is created, that identity should be used instead of the bootstrap token. | ||
|
||
This controller will also delete tokens that are past their expiration time. | ||
The binding of `system:bootstrappers` to the ability to submit certs is not part of the default RBAC configuration. Tools like `kubeadm` will have to explicitly create this binding. | ||
|
||
<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> | ||
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/super-simple-discovery-api.md?pixel)]() | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
MUST NOT
feels wrong - how can we lock down the discovery endpoint? Shouldn't this be up to RBAC policy?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just for "cluster info" stuff. (see the section defining that) Ways to identify where the cluster is and the CA bundle. This is generally ~public information and so is stored and (with the
--discovery-url
flag can be served in something like a gist if users choose.Because this isn't often secured, we are specifying that you MUST NOT put credentials in there.
My ideal (and a previous iteration of this design) would be to not re-use the kubeconfig format and define something that just has cluster location information. But we agreed on not inventing a new config file format that overlaps the current kubeconfig.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It feels very odd to prevent people from doing something more secure than what you imagine.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you misunderstand.
If you already have credentials and a kubeconfig you don't need to use the bootstrap mechanism at all. We'll be able to skip that phase.
But if you don't already have a kubeconfig kubeadm supports the following flow:
This is about how we do those. One method of doing (1) above is to hit a well known https endpoint. In that case, if users accidentally upload a kubeconfig with credentials then they are exposing those more widely than they should.