Update bootstrap design doc with kubeadm UX

[jbeda]

Signed-off-by: Joe Beda joe.github@bedafamily.com

[dmmcquay]

Add description (only flag missing one).

[jbeda]

Added more semantics here.

[luxas]

Please be super-specific on the . separator here!!

Also mention kubernetes/client-go#114

And include what we have to do with RBAC: kubernetes/kubeadm#169

And that the system:bootstrappers Group should be granted access to the Certs API

[luxas]

I don't think we'll have time to impl this

[luxas]

in v1.6

[mikedanese]

This feature could be added without breaking compatibility

[luxas]

I actually don't think we have to support this, do you?
If the user wants a TTL, they can remove the token automatically and add a new one that suits their needs.

[luxas]

Don't want to get too heavy on the flags here

[jbeda]

2 reasons for this:

1. The "never expire" default is a bit on the sketchy side from a security point of view. Having an option from the start is a small way to start mitigating it.
2. Even having this parameter there with a help string that says "forever" will clue people in to what the default behavior is. This is a cheap way to document something that is somewhat security sensitive.

[luxas]

This doesn't make sense I think. We should just do the --token on the init side, I think this bloats the options here, since everything here's configurable with kubeadm token afterwards

[luxas]

--bootstrap-auth-token should be --tls-bootstrap-auth-token instead on kubeadm join and will work if there's a such secret with usage-bootstrap-authentication

[jbeda]

So -- do we want to support having a different token for different usages? I can see going down to a single token but I want to make sure we don't preclude scenarios here.

[jbeda]

I'll remove the specialized flags. We can look to add them back later.

[luxas]

I don't think this is needed. I've implemented this as a client.CoreV1().ConfigMaps(metav1.NamespacePublic).Get(bootstrapapi.ClusterInfoConfigMap) and I don't think this path should be configurable.
If the user needs to config this, go with https or file

[jbeda]

I'm OK removing this for now. The only situation I can think of, at some point, may involve federation type stuff where I use one cluster to discover another cluster. But I'm having a hard time really wrapping my head around those scenarios.

[luxas]

Point out the signing makes is valid for --discovery-token and authentication makes it valid for --tls-bootstrap-token

[luxas]

this is a duration, point that out

[luxas]

nit: token-id.token-secret
So you mean this should be like this with both the full expiration time and the time that's left from Now()?

TOKEN                                   TTL    EXPIRATION                       USAGES
abcdef.0123456789abcdef      8h      01.01.2018T00:00:00.000   signing,authentication

or just

TOKEN                                   TTL    USAGES
abcdef.0123456789abcdef      8h     signing,authentication

[jbeda]

I was thinking listing the expiration time in addition to the TTL. 2 reasons -- TTL will be rounded to be human readable and it doesn't hold up if someone is looking at the output from a script.

But up to you.

[jbeda]

@luxas @dmmcquay PTAL -- I think I responded to all the feedback. Let me know if I missed anything.

[dmmcquay]

lgtm, small spelling error.

[dmmcquay]

spelling (leoading)

[mikedanese]

I'm of the opinion that this should not support http or or https+insecure. It doesn't today.

[jbeda]

Sure -- happy to remove it. We can always add it back later if we see a compelling case.

[jbeda]

@mikedanese @dmmcquay -- PTAL

[luxas]

Great progress made here!
Some things yet to fix though...

[luxas]

This token is used for both discovery and TLS bootstrap.

suggest: This token has both the usage-bootstrap-signing and usage-bootstrap-authentication flags set to true, which means it will be used for both discovery and TLS bootstrap.

[luxas]

Hmm, can we rename these to --tls-bootstrap-from-url or --tls-bootstrap-url instead?
I think that makes more sense since what's actually happening behind the scenes is just the TLS Bootstrap, nothing else...

[jbeda]

This is discovery, not tls bootstrap. Discussion below.

[luxas]

Same goes for this arg: can it be --tls-bootstrap-from-file?

[luxas]

Sorry if I was unclear last review (I was in a hurry), but I do think we should keep what you had in terms of exposing both --discovery-token and --tls-bootstrap-token and using --token as a shorthand for setting both.

This way, the only --discovery-xxx flag we have is token, because that's actually the only one that's gonna do discovery with the cluster-info ConfigMap, JWS validation and all that. I see this as two phases:

1. Unsecure HTTP GET call to the cluster-info ConfigMap for getting the CA and master list and validate the content we recieved using the token -- this is discovery and is only available for the token method
2. TLS Bootstrap -- do the CSR dance and connect to the API Server with whatever creds you have (existing client cert or token), but this is most often a token regardless of method -- this phase is available for file, url and token.

I propose to define this in the doc and refer to 1. as discovery and 2. as just TLS bootstrap.
In the future, 2. may also be passed to the kubelet: kubernetes/kubeadm#173

So TL;DR; --discovery-token is used in 1. and --tls-bootstrap-token is used in 2. and --token sets both

[jbeda]

I'm confused here.

Discovery == "getting and trusting the cluster-info file". We have 3 ways to do this: token, HTTPS endpoint (trusting the system CA bundle) and local file. This is driven by the --discovery-* flags.

I think you are wrong wrt 2. The TLS bootstrap dance only works with a token. It is what is used to authenticate to the API server using temporary credentials (the token) in order to submit a CSR. That would be the --tls-bootstrap-token flag. I'll make sure we have that.

[luxas]

Ok, I see how you're thinking.
Let's see if we can come to a conclusion here... This is what I thought when I wrote that comment:

Discovery

Scenario A: We only have a token and an address to an API Server. What do we do?
We still need a CA cert, and in the future the addresses to the rest of the API Servers in the clusters.

We construct a KubeConfig file like this:

apiVersion: v1
kind: Config
clusters:
- cluster:
     server: https://BOOTSTRAP_SERVER:PORT
     insecureTLSVerify: false
  name: discovery
contexts:
- context:
    cluster: discovery
    user: ""
  name: discovery
current-context: discovery

and do

client.CoreV1().ConfigMaps(metav1.NamespacePublic).Get("cluster-info")

in order to get the KubeConfig with the CA cert and cluster addresses + the JWS token.
Then we validate the JWS token against our own token and the KubeConfig file we've got.

If everything succeeds, we'll end up with a KubeConfig file like this:

apiVersion: v1
kind: Config
clusters:
- cluster:
     server: https://SERVER:PORT{https://SERVER2:PORT2,...}
     certificate-authority-data: <cacertbytes>
  name: tlsbootstrap-ready
users:
- name: tlsbootstrap-ready
  user:
    token: <tls-bootstrap-token-id>:<tls-bootstrap-token-secret>
contexts:
- context:
    cluster: tlsbootstrap-ready
    user: tlsbootstrap-ready
  name: tlsbootstrap-ready
current-context: tlsbootstrap-ready

This is what we need for the next phase (TLS Bootstrap). I.e. this is the output of the discovery phase and the input to the TLS bootstrap phase in my world.

Scenario B: We only have a KubeConfig file with credentials that we can use to talk in a secure manner to the cluster. What do we do?

Well, since we have all the information needed for TLS Bootstrap already, we'll just skip this phase and go on to the next.

At least, this is what we do now in kubeadm. I don't know, maybe it makes sense to hit the cluster-info ConfigMap regardless in order to check for updated information (which can be more than the CA and the apiserver list in the future). If that was what you thought, then I support that. But FYI, that's far from what's implemented now.

TLS Bootstrap

This is just "the normal" TLS bootstrap flow, which in the future may/should be implemented by the kubelet itself for easier lifecycle management of those certs.

The only thing we need as an input here is a KubeConfig file that has:

• at least one address to an apiserver
• a CA cert so we can trust the apiserver in question
• an user that has access to the Certificates API. Using a token here is by far the most common method, but I don't think it matters how you're connecting. By default in kubeadm, the system:bootstrappers group is granted access to the Certs API. If the cluster admin decided to make client certs that has system:bootstrapper as its Organization, I guess this will just work as well as using a token.

If we should hit the cluster-info endpoint regardless of (token|file|https), then there should be --discovery-(file|token|https). If we should just do a "fast-forward" and use the kubeconfig taken from the file or https and give it to TLS bootstrap directly (how it's implemented currently), this should be --tls-bootstrap-(https|file)

Anyway I think it's very important to define the inputs/outputs here. WDYT about this?

[jbeda]

Wrt Discovery -- This is all outlined in the first section of the document. This is about distributing information about the cluster itself including the CA bundle for that server. Over time we can use this to find a set of API servers but that is for the future. This has nothing to do with auth at all and it is bad news if you put TLS client certificates in cluster-info. Earlier in the doc I say "It is expected that the kubeconfig file will have a single unnamed Cluster entry. Other information (especially authentication secrets) must be omitted."

But, once you find a cluster, how do you get a unique client certificate signed by a CA that the server trusts? That is the TLS bootstrap flow. And the option we are using here os to also use the token for that. If you have a client certificate that you've already gotten via other means we should support that. But that really isn't "bootstrap" in any way.

The client TLS stuff is a bit out of scope for this doc but I'd be cool with a set of flags like this:

• --client-tls-bootstrap-token Configure client TLS by using a token and the Cert API
• --client-tls-certificate and --client-tls-key Instead of doing TLS bootstrapping with a token, read the client certificate and key from these files. Install them in the appropriate directories. Verify that they can be used to talk to the server. Note that the flag names mirror the kubectl flags somewhat.

If either --client-tls-certificate or --client-tls-key are set then it is assumed that users do not want to use the cert API to bootstrap TLS. If both are not set or they cannot successfully be used to communicate with the server then an error is thrown.

[luxas]

This has nothing to do with auth at all and it is bad news if you put TLS client certificates in cluster-info

Sorry, that was not what I meant. I meant that you could make a TLS-Bootstrap-only client cert (instead of a token) in your KubeConfig file that will be used for TLS bootstrapping i.e. in the second phase of what kubeadm does. I just said that I think this is possible as an answer to:

I think you are wrong wrt 2. The TLS bootstrap dance only works with a token.

--client-tls-certificate and --client-tls-key Instead of doing TLS bootstrapping with a token, read the client certificate and key from these files. Install them in the appropriate directories. Verify that they can be used to talk to the server. Note that the flag names mirror the kubectl flags somewhat

I agree those flags would be really cool.

[luxas]

I think kubeadm token create [token] [flags] is much more UX-friendly. If [token] isn't specified, it will be autogenerated. Sounds good?

[jbeda]

sure

[luxas]

Everything here is UTC time I suppose?

[jbeda]

Since TTL is "time from now" UTC doesn't come in to play. This will be converted into an absolute UTC time as it is written in to the token secret.

[luxas]

specify the id or the secret

specify the token id or the full token

[luxas]

Initially, we're only opening get configmap in kube-public to avoid getting opening ourselves to the world too much

[jbeda]

There is nothing else in kube-public. The whole idea is that it will be totally public. But, sure.

[luxas]

In kubeadm, this defaults to true

[jbeda]

This is for secret itself, not the UX for kubeadm. If a usage is missing from the token secret, users should assume that usage isn't allowed. I'll make it more clear in the text.

[luxas]

Defaults to true in kubeadm

[luxas]

We should also point out that access to a bootstrap token equals root access to the cluster if --insecure-experimental-approve-all-kubelet-csrs-for-group=system:bootstrappers.

I'm planning to propose a solution to that problem (probably a new API Object to the Certs API Group if other things doesn't work, sigh)

[liggitt]

We should also point out that access to a bootstrap token equals root access to the cluster if --insecure-experimental-approve-all-kubelet-csrs-for-group=system:bootstrappers.

Pretty sure that's "working as designed" until we partition node permissions.

I'm planning to propose a solution to that problem (probably a new API Object to the Certs API Group if other things doesn't work, sigh)

To accomplish what, dynamically enabling/disabling that setting? I have on my list to replace the hard coded "always approve" group with a permission check in the controller manager using a subject access review ("did the user that submitted this CSR for a kubelet client cert have permission to 'join' 'nodes' named 'mynode'", etc)

[jbeda]

Added a comment saying that our documentation should make it clear how sensitive this token is.

[luxas]

Ok, so now I see what confused me a little:

• The current behaviour of kubeadm at HEAD -- totally different than described here
• The fact the content in the file/https kubeconfig is sufficent for doing the TLS Bootstrap straight away. I thought we could just go on and do that. However, it is preferable to hit the cluster-info ConfigMap in any case, I agree with this. Maybe you could clearly point out that when using the out-of-band (file) or https method, the implementation should still talk with the cluster-info endpoint?

However, I think that it should be stated that when using https/file, the implementation should (must?) omit the user info when getting the information from the cluster-info ConfigMap in order to not unnecessarily send auth data when it's not necessary (this is public info after all).

Also, in this case of https/file, the communication between the implementation (kubeadm) and the API Server MUST BE secure with https.

[luxas]

You haven't defined ClusterInfo. I think you just mean the KubeConfig file with the list of addresses to API Servers with the cluster-info ConfigMap exposed and the CA cert(s)

[jbeda]

It is defined in the section called "Cluster location information" but isn't as clear as it could be. I'll add more text there and call out the term.

[luxas]

Pretty sure that's "working as designed" until we partition node permissions.

@liggitt Indeed "by design", but I don't think our users realize this really

At a first glance, would a policy like this make sense to you?

kind: CSRApproverBinding
apiVersion: certificates.k8s.io/v1beta2
metadata:
  name: my-binding-between-generic-tokens-and-kubelet-privs
subject: // The one that posted the CSR
  kind: Group // Can also be User or ServiceAccount
  name: system:bootstrappers
allowedAttributes: // Allowed attributes that the CSR can include for the controller-manager to auto-approve
  groups:
  - system:nodes
  - system:basic-user
  users:
  - some-misc-user
  - some-other-allowed-user-to-claim
  // any other attributes needed here?

or it could be

kind: CSRApproverBinding
apiVersion: certificates.k8s.io/v1beta2
metadata:
  name: my-approval-of-new-master-privs-with-token-abcdef
subject: // The one that posted the CSR
  kind: User
  name: system:bootstrapper:abcdef
allowedAttributes: // Allowed attributes that the CSR can include for the controller-manager to auto-approve
  groups:
  - system:ḿasters
  users:
  - kube-apiserver

This way we somehow could limit the maximum damage that can be made if a token is exploited.

This was just the first thing I came to think about... feel free to shoot it down :)

Anyway, I'd want a mapping between a Group/User/SA and the maximum allowed attributes it should be able to request for the controller-manager to auto-approve. This way we could make different kinds of tokens that can ask for different kinds of privs. We also achieve a pretty good balance between security and convenience (we don't have to approve the CSRs by hand). WDYT?

(Sorry, I know this isn't exactly what's discussed in the PR, but it seems quite related and since we started talking about this already...)

[jbeda]

Pushed some more changes. PTAL. Hopefully this clears up some of the confusion. I'm really trying to keep bootstrapping the server ID separate from TLS bootstrap. While we want to use the same token for both for ease of use, they are really pretty separate.

[luxas]

👍 it gets clearer and clearer.
The last thing I'd like here is an explanation why we should copy the ClusterInfo kubeconfig out-of-band to a node just to fetch the ClusterInfo object once again (via HTTPS).

Something like:
When using the HTTPS or File method, the implementation should securely (via HTTPS) contact the API Server(s) that exposes the public ClusterInfo object once again in order to refresh the credentials. Otherwise, the out-of-band ClusterInfo object could get out of sync with the latest information stored in the ConfigMap. Imagine this situation:

Information in the ClusterInfo ConfigMap in API Server named apiserver1

... the rest of the kubeconfig object...
cluster:
  servers: apiserver1:6443
  certificate-authorithy-data: cacert1

Then, out-of-band, this information is transferred to a node in file my-discovery.yaml.

One day after, the publicly-available ClusterInfo kubeconfig is changed to:

... the rest of the kubeconfig object...
cluster:
  servers: apiserver1:6443,apiserver2:6443
  certificate-authorithy-data: cacert1,cacert2

and if the implementation (e.g. kubeadm) didn't refresh this information by connecting with my-discovery.yaml and then updating its cached data, the node would never know that a new apiserver and ca cert have been added to the cluster.

Something like that, just to make it clear that the client MUST refresh the data also when using the HTTPS/File method, because the information might be stale.

[luxas]

controller-manager

[jbeda]

Right now the ClusterInfo is pretty much just a single address and the CA bundle. There is no real way to rotate certificates without notifying everyone out of band. I think it is okay for kubeadm to update stuff from the cluster-info configmap but I don't think it is a MUST. But it MUST verify the certificate chain when hitting the server. So, the only time insecure HTTP should be used is when we have the token to verify the contents of the payload.

[luxas]

I think it is okay for kubeadm to update stuff from the cluster-info configmap but I don't think it is a MUST

Fair enough. With HA and kubeconfig v2 this becomes a MUST though, but I'm fine with defining it generally as a MAY.

+100 to adding the section that describes that the implementation MUST verify that the CA cert for the given server is valid, I was missing that from the original proposal.

This is fine now I think, we should probably mention how to hook up this design with the next generation of KubeConfig v2 (i.e. how do we do the upgrade, what will the constants for be, etc.), but that can wait.

[luxas]

Please squash though 😉

[jbeda]

OK! Squashed. I think that we are in good shape but I'd like you to take one more pass before it gets merged. I'd also like to give @mikedanese a chance to give it another look if he wants.

[luxas]

One legit idea re description and one crazy one with cluster-role...

I'd also want this spec to state how to deal with duplicate tokens. What happens if:

• two tokens have the same id, but different content (secrets for instance) -- this one is nasty
• two tokens have exactly the same content?

[luxas]

Hmm, I wonder whether we actually should add a well-known description field here?
Don't wanna bloat the spec, but as we grow the responsibilities of these tokens; we might for example want to allocate a token for the scheduler and the controller-manager (kubernetes/kubeadm#172) respectively, and then it would probably be helpful with a human-friendly message that states "this token is for the scheduler" and if the user goes and removes it, the scheduler will lose access to the apiserver.

In any case, when you have 15 different tokens with different TTLs and usages and a new person comes and looks at it, he/she will have no idea where they are used and what they are used for. Adding a generic description field to this spec and to kubeadm token list will clarify things a lot. It also aligns with the rest of k8s where the metadata field is constantly present. WDYT?

[jbeda]

I'm in favor of generic free form description fields. I'm happy to add it. It is a bit of an overload -- this could go in an annotation or it could go in the of the secret.

[luxas]

I just think it would be really helpful for kubeadm token list to have a DESCRIPTION column.
I'm open to using a metadata field on the Secret as well, but it feels like we're better of being consistent with the rest of the data

[luxas]

I don't want to bloat RBAC into this spec either, but would it make sense to add a --cluster-role to this command?
This could essentially be a shorthand for

kubectl create clusterrolebinding bootstrap-token-(token-id) --cluster-role (cluster-role) --user system:bootstrap:(token-id)

that would be applied in the same go if this arg is specified. This way the user could easily attach an extra role...

Well, feel free to ignore this, it was just a crazy idea that sprung to mind :)

[jbeda]

I don't think we want to encourage people to use these tokens outside of just bootstrapping. We can always add it later if we find another bootstrap-ish use for it, but, for now, I'd like to keep these scoped.

[luxas]

I'm ok with that

[jbeda]

Wrt duplicate tokesn -- in the bootstrap signer I just ignore one of them and warn. See https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/bootstrap/bootstrapsigner.go#L280.

I'm also happy to say something like "if two secrets have the same token ID then that is considered a corrupt configuration and all of the secrets with that ID are ignored". That might be safer and more reliable but will require changes to the bootstrap signer.

[luxas]

I'm in favor of marking a duplicated token invalid by all consumers. Please add that to the doc

[luxas]

I think it might make sense to output the Secret name in the first column here actually.
Otherwise you'll have a very hard time knowing which token matches which Secret without iterating all of them

[jbeda]

I'd like to avoid that. Most consumers will be unaware that the tokens are implemented as kube-system secrets under the covers. It will be confusing and break the mental model for users at this point.

Instead, we could warn loudly if we have duplicates and list all the secret names. Perhaps with instructions on how to use kubectl to delete them all and recreate the secret.

[luxas]

It will be confusing and break the mental model for users at this point.

Fair point

[luxas]

I'm ok with that

[luxas]

I just think it would be really helpful for kubeadm token list to have a DESCRIPTION column.
I'm open to using a metadata field on the Secret as well, but it feels like we're better of being consistent with the rest of the data

[jbeda]

A third option for duplicate tokens -- make the secret name a MUST and eliminate the possibility. Warn of all secrets with the bootstrap token type that don't match up.

[luxas]

A third option for duplicate tokens -- make the secret name a MUST and eliminate the possibility. Warn of all secrets with the bootstrap token type that don't match up.

IMO, this is the best alternative. We'll restrict bootstrap tokens to be named bootstrap-token-(token-id) and the TokenCleaner might (if we agree on that) remove all Secrets that doesn't follow that pattern.
The TokenCleaner can also remove bootstrap tokens that are invalid (e.g. bootstrap-token-abcdef and token fedcba.xxxx...)

Then the listing becomes so much easier, you can just rely on bootstrapapi.BootstrapTokenSecretPrefix+tokenid for getting the secret. With the current impl. we have to loop all the tokens every time we touch them.

(In any case, kubeadm token create should error out if there already is a token with the same id.)

[jbeda]

Ok -- I'll write this. But I'm not in favor of the TokenCleaner removing those corrupt tokens. This may lead to the case where someone writes something to the cluster and before they can debug it just goes away.

We can always add that back in later.

Ok -- I'll update the spec and start a delta on the BootstrapSigner.

[luxas]

But I'm not in favor of the TokenCleaner removing those corrupt tokens.

Yup, just a suggestion. But I'd at least like the TokenCleaner to annotate or label it "invalid" in some way.

[jbeda]

PTAL -- I'm going to hold off on having the TokenCleaner annotate the secret as invalid at this point. This is a new write path in that secret and the chances of this happening are pretty slim -- someone has to be managing secrets outside of kubeadm token ... and screw it up. Just logging in the KCM should be enough here.

[justinsb]

Is the plan to move this to an API type? Can we do it now?

[jbeda]

I wanted to start as an API type from the start but after a lot of discussion over many months we settled on this. You can read the super long gory history here: kubernetes/kubernetes#30707

[justinsb]

I agree that we should make tokens dynamic. But I am unclear why bootstrap tokens are different from "normal" auth tokens?

[jbeda]

Normal auth tokens are defined in a CSV file on disk. We are scoping this mechanism to just bootstrap as we are storing the tokens in etcd. The general recommended approach is, for day to day usage, use TLS client certs or another more full featured auth provider.

[justinsb]

MUST NOT feels wrong - how can we lock down the discovery endpoint? Shouldn't this be up to RBAC policy?

[jbeda]

This is just for "cluster info" stuff. (see the section defining that) Ways to identify where the cluster is and the CA bundle. This is generally ~public information and so is stored and (with the --discovery-url flag can be served in something like a gist if users choose.

Because this isn't often secured, we are specifying that you MUST NOT put credentials in there.

My ideal (and a previous iteration of this design) would be to not re-use the kubeconfig format and define something that just has cluster location information. But we agreed on not inventing a new config file format that overlaps the current kubeconfig.

[justinsb]

It feels very odd to prevent people from doing something more secure than what you imagine.

[jbeda]

I think you misunderstand.

If you already have credentials and a kubeconfig you don't need to use the bootstrap mechanism at all. We'll be able to skip that phase.

But if you don't already have a kubeconfig kubeadm supports the following flow:

1. Discovery where the cluster is and what root CA bundles to use
2. Use some temporary credentials to submit a CSR to the cert API

This is about how we do those. One method of doing (1) above is to hit a well known https endpoint. In that case, if users accidentally upload a kubeconfig with credentials then they are exposing those more widely than they should.

[justinsb]

(Commenting here as github won't let me comment on the other lines)

• Implementation note: the API server doesn't have to expose a new and special insecure HTTP endpoint.
• kubeadm requests a ConfigMap containing the kubeconfig file defined above.
  • This ConfigMap exists at a well known URL: https://<server>/api/v1/namespaces/kube-public/configmaps/cluster-info
  • This ConfigMap is really public. Users don't need to authenticate to read this ConfigMap. In fact, the client MUST NOT use a bearer token here as we don't trust this endpoint yet.

This is not a good idea. Unauthenticated access to the same endpoint always evolves into a security vulnerability.

I presume this is not enabled by default, or that it can be turned off? Have the k8s security folk signed off on this?

[jbeda]

Yes -- approved and in the merge queue. See kubernetes/kubernetes#41281

[jbeda]

@luxas @mikedanese Last call on this one. I'm going to merge at EOD as it looks like most comments have been addressed and implementation is well under way. We talked about this at the lifecycle meeting on Tuesday too.

[luxas]

@jbeda That sounds good to me.
You don't have to do it in this PR, but long-term we should think about how to evolve the content that's exposed, which currently is a KubeConfig v1 file.

Let's imagine we invent KubeConfig v2 tomorrow -- then we should claim the kubeconfigv2 key on the cluster-info ConfigMap I think, and sign JWT tokens based on that one with prefix jwt-kubeconfigv2- or so... I guess that makes sense, right?

If a there is a server that exposes only the kubeconfigv2 key(s), then older consumer clients like kubeadm v1.6 won't be able to talk to it, but I suppose newer clients (kubeadm v1.9) will be able to talk to older servers that expose kubeconfig (v1) only, given we choose so.

Some words on that -- how we are planning to upgrade this signing stuff to something that's better at exposing information about the cluster than KubeConfig v1 might be good, but that's not a blocker for me.

Thanks for reworking this, it's much better now 👍

[luxas]

When specifying some kind of token, one also have to specify where the master is located.
Proposed flag for that --masters

[jbeda]

Github glitch made me miss this before I merged. Needed to reload page.

But I think we talked about this on slack and decided to keep the CLI flags consistent with current behavior where the master to contact with the token isn't behind a flag.

[jbeda]

Merged -- thanks for all the feedback.

As for kubeconfig v2 -- I totally agree. I'd love to evolve that forward. Part of the early negotiation around this was to not boil the ocean and start with current kubeconfig even though it might be sub-optimal.