Configurable Cluster Domain

[fen4o]

The Cluster Domain is hard-coded to cluster.local. This won't work for clusters with custom domain name in kube-dns and kubelet

Can we have it configurable via additional command line option? In kubelet right now its --cluster-domain=... and in kube-dns --domain=...

[hongchaodeng]

Right. Thanks for the feature request.

Please be patient since we would probably spend next couple of dev cycles on testing and releases.

[trunet]

will this be handled in the near future?

[hongchaodeng]

Yes.

@trunet Would you like to send a PR for this?

[trunet]

@hongchaodeng I can if you help me telling how I can get the kubernetes domain inside pkg/util/etcdutil/member.go.

[hongchaodeng]

I will handle it in next coming release

[hongchaodeng]

@fen4o @trunet @zhenyuxx @blakebarnett
For all the users who change cluster domain of k8s, mind if I ask what are the motivations for changing default cluster domain?

[hongchaodeng]

Also @ you guys to help answer above questions
@exename @nekufa @vgkowski

[blakebarnett]

We provide a VPN into the clusters, this has the benefit of making cluster DNS available to users, but it requires that the clusters have a unique domain. We would definitely not want to give up this flexibility.

For example our users can get to their service like: https://foo.namespace.svc.<cluster domain> on their machines when connected to the VPN and querying the DNS servers properly.

[xiang90]

but it requires that the clusters have a unique domain.

So you have VPN to multiple clusters and want to access services in different clusters by having different cluster domain (DNS search domain)?

[xiang90]

@fen4o

Can we have it configurable via additional command line option?

Why do you want to make it configurable rather than just relying on what Kubelet provides on each node?

[blakebarnett]

@xiang90 correct

[hongchaodeng]

@blakebarnett
Thanks!
So let me state more clearly what we are trying to do and our assumptions (in #1341):

• Pods of the same etcd cluster need to sit within the same k8s cluster.
• Etcd pods will expand DNS with the cluster domain that kubelet gives. This assumes all kubelets of the same k8s cluster have the same cluster domain.
• For cross cluster access, note that "server" cert for etcd need to include the long domain name https://<etcd-cluster>-client.namespace.svc.<cluster domain>. Usually it just needs to be https://<etcd-cluster>-client.namespace.svc

[dghubble]

Customizing the cluster domain suffix has other legitimate uses. The default cluster.local can cause issues, but we stuck with it because many projects in Kubernetes were relying on it coreos/coreos-kubernetes#658. If etcd-operator also relies on it, that would make it increasingly difficult to move away from.

[blakebarnett]

@hongchaodeng maybe it's reasonable to require a cert be provided to the operator via configmap if the clusterdomain has been changed from the default?

[xiang90]

@hongchaodeng maybe it's reasonable to require a cert be provided to the operator via configmap if the clusterdomain has been changed from the default?

You already can do this today. We just need to update some docs.

I am more interested in why some people want to supply cluster domain via a flag into etcd operator instead of relying on what is set on kubelet.

[blakebarnett]

ah, yeah that seems like an odd situation...

[hongchaodeng]

@blakebarnett
It's secret, not configmap.

See https://kubernetes.io/docs/concepts/configuration/secret/

Mounted Secrets are updated automatically

When a secret being already consumed in a volume is updated, projected keys are eventually updated as well. Kubelet is checking whether the mounted secret is fresh on every periodic sync. However, it is using its local ttl-based cache for getting the current value of the secret. As a result, the total delay from the moment when the secret is updated to the moment when new keys are projected to the pod can be as long as kubelet sync period + ttl of secrets cache in kubelet.

[blakebarnett]

Right, makes sense. Awesome, I'll try out your new patch and do this.

[fen4o]

@xiang90

Why do you want to make it configurable rather than just relying on what Kubelet provides on each node?

Exactly for the same reason as @blakebarnett - VPN in clusters. We also plan to use it for cluster federation as well.

[xiang90]

Exactly for the same reason as @blakebarnett - VPN in clusters. We also plan to use it for cluster federation as well.

For that reason, etcd can still communicate to each other relying on the cluster domain provided by Kubelet. For external communication, you can provide whatever domain name you want to TLS assets if TLS is enabled. I do not feel the flag you mentioned is needed, correct?

[fen4o]

The real main reason why we use custom domains is because users have access to the internal kube-dns and they can query services at https://foo.namespace.svc.foo.bar. This means that both kube-dns and kubelet must be configured with the same values. In our case, kubelet will have the following configuration added to every single container it starts:

$ cat /etc/resolv.conf
nameserver 10.64.0.10
search kube-system.svc.foo.bar svc.foo.bar foo.bar eu-west-1.compute.internal
options ndots:5

The etcd-operator is not working for us, because the members' IP address cannot be resolved.
etcd-1.etcd.default.svc.cluster.local is not a resolvable DNS record in our case.

If the operator is smart enough to discover the cluster domain by itself - great, otherwise it must be configrable.

[hongchaodeng]

If the operator is smart enough to discover the cluster domain by itself - great, otherwise it must be configrable.

Yeah. This is the direction to go. Kubelet will provide default search domain for its cluster domain.

[hongchaodeng]

Please try out latest operator quay.io/coreos/etcd-operator:dev and provide feedback. Thanks!

[fen4o]

Everything looks okay.

[hongchaodeng]

Cool~
We will cut a release on Friday or early next week.