-
Notifications
You must be signed in to change notification settings - Fork 741
Conversation
Let me make a conclusion attempt:
|
doc/design/dynamic_tls.md
Outdated
- Each etcd cluster has its own CA. | ||
|
||
Cons: | ||
- Inappropriate to give etcd pod access to the CA key? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Answer: yes, it's inappropriate to give the etcd pod the CA key.
I don't think we should include this as an option.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 That was my point during yesterday's meeting where we discussed this.
I don't think we should include this as an option.
SGTM
doc/design/dynamic_tls.md
Outdated
|
||
Similar to option 2, but instead of using init container, etcd operator generates certs and puts them into a secret. Then etcd operator creates an etcd pod with the secret mounted. | ||
|
||
The option is not feasible. The pod IP cannot be known before it is created. In self-hosted mode, even if we can know the node addresses, it is still a problem that the node could be deleted and new ones be added. Thus, we can’t specify the SAN until the pod is created. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In self hosted mode the Pod IP is the node IP, right? So that is known ahead of time if the etcd-operator schedules the pod itself (like the daemonset controller)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In theory, yeah. I think @hongchaodeng was worried about edge cases where the node IP suddenly changes before the pod is scheduled. I don't think this is much of a problem, though.
If Option 2 is not an option due to security concerns, I'm more of a fan of Option 3 than 1. At least for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In theory, yeah. I think @hongchaodeng was worried about edge cases where the node IP suddenly changes before the pod is scheduled. I don't think this is much of a problem, though.
That sounds like a super-extreme edge case I don't see happening. At least, I think you'll have way worse problems if you're master node IP changes that frequently. I vote for calculating it based on node IP
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think @hongchaodeng was worried about edge cases where the node IP suddenly changes before the pod is scheduled.
Does that actually happen?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's not edge case. Let me clarify the concerns:
Concern 1: Changing existing nodes
Etcd cluster running on 3 master nodes N1, N2, N3. We are going to replace N3 with N4. In etcd 3.2+, peer's cert SAN needs to include all peer addresses/IPs. Simply adding member of N4 won't be able to join N1 and N2.
Concern 2: How to know what are master nodes?
etcd operator do not know on which node the etcd pod is gonna end up until scheduler makes the decision, and thus do not know what SAN should be included.
If we want to know that, we need to include some scheduling logic in etcd operator.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
peer's cert SAN needs to include all peer addresses/IPs. Simply adding member of N4 won't be able to join N1 and N2.
Can you expand on that? How does this affect Option 3 but not Option 1? Surely you're doing the same thing, i.e. generating a cert bound to a node IP.
etcd operator do not know on which node the etcd pod is gonna end up until scheduler makes the decision, and thus do not know what SAN should be included.
I think Lucas's point is that we can infer it. The operator already has a list of master nodes, so finding the ones that do not have etcd pods on should be simple.
If the node is replaced in the seconds after the certs are generated but before the pod is scheduled, this of course won't work. But I think this is very rare.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you expand on that? How does this affect Option 3 but not Option 1?
ref: etcd-io/etcd#8797
You are right. Option 1 will definitely not work at least for current 3.2
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually I made a mistake in understanding... Sorry for the confusion.
It's the certificate presented by remote peer: https://github.com/golang/go/blob/go1.9/src/crypto/tls/common.go#L169:2
So that means as long as the certificate for each peer includes itself, it should be fine.
Ignore the first concern.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the node is replaced in the seconds after the certs are generated but before the pod is scheduled, this of course won't work. But I think this is very rare.
Yeah. We probably do not need to care about that at the beginning.
But this is automated system. And we should definitely need to handle that if we go with option 3.
doc/design/dynamic_tls.md
Outdated
Cons: | ||
- Inappropriate to give etcd pod access to the CA key? | ||
|
||
### Option 3: CA signing cert + key, etcd operator generates certs and secrets |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also questions of how clients get client certs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At least for our use case, kubeadm will generate the CA and client certs upfront before the operator is deployed. Both sets can be stored in the same dir and mounted into the operator
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry I should make it clearer. Here we are only concerned about peer-to-peer serving certs.
Client certs is a much simpler problem because client endpoint is known. I can add that later but let's solve peer certs first.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we choose any option, I will write more details about how to generate TLS assets for client, server, peer
/cc @luxas |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! I vote for option 3
|
||
Cons: | ||
- Share the same CA signing cert + key as k8s. | ||
- Requires etcd to be trusted by k8s cluster CA (probably only appropriate for self-hosted etcd). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, this is only appropriate for self-hosting. You don't want your app etcd db to be able to access the k8s main etcd.
What's the latest plan for HA control planes in kubeadm? What's the TLS plan for the API server? I think etcd should be able to support option 3, but I'd strongly prefer option 1 for self-hosted etcd bootkube clusters if we ever get there. Having everything go through the CSR endpoints seems way cleaner than the etcd-operator managing its own CA. |
That's actually a good point -- we could have different methods for self-hosted and normal app cases. As I mentioned above, option 1 isn't exclusive to option 2 & 3. So I could think of another plan:
Just to clarify, above plan is based on that consensus made for option 1. |
If Option 1 is the default for self-hosted, who controls the deployment of the auto-approver controller? I assume the operator would deploy it by default. Where is this code going to live? Does CoreOS have enough bandwidth to write and maintain another codebase/sub-project? My preference is start with Option 3 because it's simpler and actually solves a real-world use case. If we're going to support both options, I think kubeadm will use option 3. Once we know node IPs work well with a pre-generated CA, we can then add CSR support in the 1.10 dev cycle. But if folks really feel Option 1 is better to implement first, that's fine with me. Just adding my 2c. |
Probably go in etcd operator pod? |
@jamiehannaford @luxas I couldn't see any big con for either option 1 or 3. And I want to move forward. Option 1 is still a better choice to move forward because it's what other k8s components does. We just follow the convention instead of reinventing our own and it is the safest way. |
You mean like a sidecar container?
Not really, no. If it can be implemented well and covers our use-case, I think it's a good idea. It'd be great if we can get this in before the Nov 22 code freeze, but no worries if that's not realistic. If not, HA etcd will have to wait until kubeadm 1.10. |
I can also help out with this next week if you want @hongchaodeng. Just describe any outstanding work items in issues and CC me. |
@jamiehannaford I will start digging & trying details of option 1. I need to do this as a last check and also have a better understanding of potential work. After that I will write down the work items more specifically. |
Trying to summarize this thread a bit. Option 1:
Option 3:
Hinted about this above, but depending on kubeadm's "master join" processes could kubeadm just sign all the certs ahead of time? I haven't caught up with kubeadm's HA plans, but I remember something about a "master join token" in previous proposals. So: Option 4:
I think some flavor of option 3 would be good for regular etcd clusters created by the operator. The "just generate a CA for this cluster and set up TLS for me" mode. Giving the etcd-operator the cluster CA is a significant choice. The certificates API was built partially to defer the final signing to external PKI, and going around it is something to be cautious of. However, it's clear that option 1 is a lot of work and isn't going to get done before the 1.9 release. |
I have tried to quick hack on option 1 and made sure it work. A little surprise that came out is that we might need to let user pass in additional SAN for server cert. For next step, I would update this PR with plan for implementation and testing. @jamiehannaford One question. Do you want to work on this? If yes, it is fine to just implement self-hosted path and ignore normal app path. It is also fine to leave testing for us. |
@hongchaodeng Yeah, I can help out. Self-hosted path is Option 1, right? |
Correct. |
Updated with impl. and testing roadmap |
@jamiehannaford @ericchiang @luxas
|
In kubeadm, the apiserver will talk to the local etcd pod via
How is the peer CA different from the serving CA? Are you suggesting having different CAs for each? |
Peer is about etcd deployment. Deployer/ops owns peer certs.
Yes. I couldn't find any docs on this. But this is the same reason as above. |
So as an update, we are going to implement dynamic TLS for peer only. We will keep using the static TLS for client, server initially. There are other ways to generate TLS certs for client, server. But let's focus on dynamic peer TLS first. And this should get rid of dependency on hostname and unblock kubeadm work. |
5af3207
to
073bd63
Compare
Let's move forward this design doc? We will implement dynamic peer TLS first as first step and unblocks kubeadm. It's open for what is needed for client-server. |
doc/design/dynamic_tls.md
Outdated
|
||
## Alternatives | ||
|
||
### Option 2: CA signing cert + key, generate certs on init container |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This still needs to be removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure
|
||
> For initial design, we focus on self-hosted etcd use case, although the same methodology applies to app etcd. | ||
|
||
[Kubernetes certificates API][k8s_csr] provides the ability to send and approve CSR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This case was covered in the bootkube auto-TLS proposal. Might want to link to that since it talks about this case a little more in depth?
Fixed. |
lgtm, feel free to merge |
Does it mean we cannot use the same for etcd as an app, not a kubernetes usecase? |
@raoofm self-hosted etcd and etcd clusters running in the cluster have different TLS requirements because their networking and trust domain is different. For self-hosted etcd, which runs in host networking, you have to use the node addresses (even as new nodes are added), and it has to be trusted by the API server. For clusters running on top of kubernetes, the operator can know the service DNS names before it deploys the individual pods. The etcd cluster's certs only have to be trusted by the apps that want to talk to it. So whatever solution the operator goes with will likely be aimed at one of these use cases initially. We can expand it to the other use case later. |
@raoofm |
[skip ci]
ref: #1617, #1465