Implement support for Kubelet Dynamic Configuration

[mikedanese]

From @luxas on October 5, 2016 19:26

We should use Kubelet Dynamic Settings by pushing values given to kubeadm init to the apiserver when started, and that will make all kubelets in the cluster reload and pick up those configuration values.

That's much better than having to modify every 10-kubeadm.conf systemd dropin file in the whole cluster.

Copied from original issue: kubernetes/kubernetes#34132

[mikedanese]

From @luxas on October 5, 2016 19:28

cc @kubernetes/sig-cluster-lifecycle

[mikedanese]

From @errordeveloper on October 5, 2016 20:25

Is this feature already usable?

[mikedanese]

From @luxas on October 5, 2016 20:29

I don't know actually, but I think an alpha version of it exists

cc @mtaufen @vishh @mikedanese @dchen1107 @yujuhong

[mikedanese]

From @axsuul on October 7, 2016 6:58

Is this related to customizing arguments for the kubernetes API server? I'm trying to customize the secure port it listens on (--secure-port=8081) but I'm not sure how to do it when setting up the cluster with kubeadm.

[mikedanese]

From @errordeveloper on October 7, 2016 7:15

James, no this is not related. To your question, please try editing
/etc/kubernetes/manifests/kube-apiserver.json, but please create another
issue if you have more questions.

On Fri, 7 Oct 2016, 08:59 James Hu, notifications@github.com wrote:

Is this related to customizing arguments for the kubernetes API server?
I'm trying to customize the secure port it listens on (--secure-port=8081)
but I'm not sure how to do it when setting up the cluster with kubeadm.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
kubernetes/kubernetes#34132 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPWSx8wMyYVNJ49UBPb1y69d4nOSja-ks5qxe24gaJpZM4KPMP5
.

[mikedanese]

From @axsuul on October 7, 2016 7:58

@errordeveloper Thanks, I've opened up issue #34307 with more questions

[mikedanese]

From @mtaufen on October 7, 2016 17:3

There is an alpha version of the dynamic settings feature, but it still needs a lot of work. I'm going to be mostly focused on GCI-related things for v1.5, but am planning be giving it high-priority focus for v1.6. I wouldn't recommend building on top of dynamic settings just yet, as it will undergo some significant changes (e.g. we will move to telling a node which configmap to use via annotations rather than creating configmaps with a per-node naming convention).

As it stands you would have to create a configmap for every node in the cluster anyway, and based on @luxas's first comment in this issue, it looks like you are trying to avoid per-node work, so that might be another reason to wait. Granted, this per-node work is still probably less effort than modifying systemd conf across a cluster...

[pires]

#64 and #42 are potential use-cases for this.

[luxas]

We are blocked on this one from other persons/sigs: kubernetes/kubernetes#27980

[jamiehannaford]

This might be possible for v1.8.
Feature issue: kubernetes/enhancements#281
alpha PR in review: kubernetes/kubernetes#46254

[luxas]

@mtaufen Will the alpha version of Kubelet Dynamic Configuration be easily upgradeable?
I mean, will it be of beta quality but alpha just because of the ladder and policy to not just enable things by default?

kubeadm could be a good way to test these things out on real clusters during the cycle and then we can decide whether we want to enable it or not in v1.8

[mtaufen]

It depends on how confident we are that the API types for the Kubelet won't change after 1.8. It's too early to say right now.

[luxas]

We have some things to fix up yet:

1. Fix a panic Panic when DynamicKubeletConfig is enabled. #553 (Fix panic when assigning configmap UID of kubelet configuration. kubernetes#56008)
2. ClusterDNS should be set conditionally based on cfg.Networking.ServiceSubnet (KubeletConfiguration.BaseConfig.ClusterDNS defaults to the tenth address of MasterConfiguration.Networking.ServiceSubnet kubernetes#56013)
3. Re-engineer the kubeadm join logic to wait for the kubelet to create /etc/kubernetes/kubelet.conf and use those credentials to PATCH the node
4. Write the the marshalled kubeletconfig object to /var/lib/kubelet/config/init/kubelet so that the kubelet will start up with the right params on init/join. The only params expected in the kubelet command-line after this is kubelet --init-config-dir /var/lib/kubelet/config/init --dynamic-config-dir /var/lib/kubelet/config/dynamic

[mtaufen]

FYI --init-config-dir is likely to change a bit, wrt this thread: kubernetes/kubernetes#55718 (comment)

[luxas]

Yeah, we'll accommodate accordingly when we move out of alpha with this on the kubeadm side.
Thanks for keeping us updated @mtaufen

[luxas]

FYI @timothysc ^
We'll do a revised version of https://github.com/kubernetes/kubeadm/blob/master/docs/design/design_v1.8.md soon-ish (with @fabriziopandini?) and include this in there.
Everything here is alpha gated in v1.9, planned for beta in v1.10

[luxas]

I think all the tasks needed to be performed were merged last week, so closing this as fixed for an alpha state now. The design of this feature is covered in design doc update of v1.9 #573.

We'll open new issues later for graduating this feature to beta on the kubeadm side.

Thanks @xiangpengzhao for the awesome work on this 🎉!

[bronger]

Is #83 supposed to work for v1.11? Because I still need to set

KUBELET_EXTRA_ARGS=--cluster-dns=10.67.97.10

in /etc/sysconfig/kubelet so that the IP of the DNS server in the pods is correct. I called:

kubeadm init --apiserver-cert-extra-sans kubmaster,kubmaster.mygreatdomain.de \
    --service-cidr 10.67.97.0/24

Without setting KUBELET_EXTRA_ARGS, the DNS is looked for at 10.96.0.10.

[neolit123]

@bronger
i cannot comment on the topic if the kubeadm flag (service-cidr) should somehow translate to the DNS IP in the kubelet.

but assuming you have transitioned to the kubeadm config v1alpha2 you can use the following:

nodeRegistration:
  kubeletExtraArgs:
    cluster-dns: 10.67.97.10

this will help you to not have to update the kubelet file.

ref: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/

[luxas]

@bronger check the /var/lib/kubelet/config.yaml file, cluster-dns should be set correctly there in v1.11

[bronger]

@luxas No, it says

clusterDNS:
- 10.96.0.10

[luxas]

@bronger please open a new issue with all the required information to debug that. Might be a bug.

[zerkms]

@bronger have you reported one? could you refer to it please, I can confirm on a latest stable kubeadm kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:14:39Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} the problem still is reproducible.

[bronger]

No, I haven't so far. Long vacation and business trip came into way. @luxas Does it make sense to just re-open #83?

[zerkms]

I think #83 perfectly fits, the new one would be identical