Pods can be scheduled in new masters between kubeadm init and master taint

[yagonobre]

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  AWS
• OS (e.g. from /etc/os-release):

Debian GNU/Linux 9 (stretch)

• Kernel (e.g. uname -a):

Linux  4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux

What happened?

If you have multiple masters, and a daemonset, when you register a new master the daemonset will scheduler a pod in this master before the master get tainted

What you expected to happen?

Pod without toleration to master taint, not get scheduled on master.

How to reproduce it (as minimally and precisely as possible)?

• Create a etcd cluster
• Create a kubernetes cluster pointing to external etcd
• Create a daemonset
• Add a new master to cluster

Anything else we need to know?

Put the taint in the kubelet config of master can solve that, also I can work on this issue.

[neolit123]

hello and thanks for the report,

Put the taint in the kubelet config of master can solve that, also I can work on this issue.

before sending a PR, can you point to the source code changes that you would like to make to get this working - e.g. source code links + a short explanation?

[yagonobre]

@neolit123 sorry for the delay, I took a time to get a look on the kubeadm code

If I remove registerTaintsUsingFlags from kubeletphase.WriteKubeletDynamicEnvFile and make kubeletphase.buildKubeletArgMap always write --register-with-taints (See kubeletphase). I just need to remove the taint step from MarkMaster phase. ( Mark Master )

[neolit123]

this is an interesting situation.
you are using v1.10.2 too which is quite outdated with our current codebase and it's unlikely that we can backport the change for older k8s versions.

this is linked to this comment for master nodes:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/cmd/init.go#L289-L294

for worker nodes we already do always set registerTaintsUsingFlags to true:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/cmd/join.go#L488-L489

for master nodes we never do it, but later mark the master as one of the final phases:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/cmd/init.go#L419-L420

note that this also sets labels:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/markmaster/markmaster.go#L57

if we always do registerTaintsUsingFlags == true for WriteKubeletDynamicEnvFile() this arguably defeats the purpose of MarkMaster() unless we also take into account the recent addition of --join-control-plane (AKA --join-master) support and also labels:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/cmd/join.go#L346-L348

in my opinion having the taints in the kubelet phases makes sense and also solves the issue for you but we have to talk with others about this.

/assign @luxas
/assign @fabriziopandini
/cc @kubernetes/sig-cluster-lifecycle-bugs

[yagonobre]

Thanks @neolit123, no problem about not backport this. And yes, MarkMaster() will just care about labels.

[timothysc]

in 1.12 you can provide an initial taint on join and control-plane-join

[blurpy]

@timothysc How do you provide an initial taint on control-plane-join? I'm seeing this exact issue when joining new masters in a 1.15 cluster.

I can't see any relevant parameters in the documentation for adding taints, and it's not possible to join using a config file:

# kubeadm join --control-plane --config /etc/kubernetes/kubeadm-master-config.yaml
     can not mix '--config' with arguments [control-plane]

Anyway, even if it did work, this seems like a workaround and not a fix. Is it not possible to taint the master at an earlier stage automatically? Seems like @neolit123 is having some ideas.

[neolit123]

@blurpy we are tracking the same problem here i think: #1621