kubeadm join on slave node fails preflight checks

[mikedanese]

From @ndtreviv on November 17, 2016 12:13

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No.

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

• kubectl join
• preflight
• kubelet is not empty

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:
  AWS EC2 Classic ami-45b69e52 m3.2xlarge instance type in us-east-1 region.

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.1 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Kernel (e.g. uname -a):

Linux ip-10-159-146-253 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:
  N/A
• Others:
  N/A

What happened:
I was standing up a 3-node cluster following the instructions here: http://kubernetes.io/docs/getting-started-guides/kubeadm/

Commands I ran on each slave:

sudo su -
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y docker.io
apt-get install -y kubelet kubeadm kubectl kubernetes-cni

Additional commands run on the master:

kubeadm init --pod-network-cidr=10.244.0.0/16
kubectl apply -f flannel.yml

flannel.yml:

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "type": "flannel",
      "delegate": {
        "isDefaultGateway": true
      }
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: amd64
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel-git:v0.6.1-28-g5dde68d-amd64
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      - name: install-cni
        image: quay.io/coreos/flannel-git:v0.6.1-28-g5dde68d-amd64
        command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

ssh-d onto slave node and ran - as per the documentation - (anonymised to protect the not-so-innocent):

sudo su - 
kubeadm join --token=[TOKEN] [MASTER IP]

Output was:

Running pre-flight checks
preflight check errors:
	/var/lib/kubelet is not empty

What you expected to happen:
I expected the node to join the cluster and output data according to the docs: http://kubernetes.io/docs/getting-started-guides/kubeadm/

As I understand the architecture, each node has to have a kubelet on it. I have a feeling that some preflight checks have been added for something else, reused for this and aren't quite appropriate. I may be wrong, though!

How to reproduce it (as minimally and precisely as possible):
Create three AWS EC2 Classic ami-45b69e52 m3.2xlarge instance type in us-east-1 region.
Run the commands as I did.

Anything else do we need to know:
No.

Copied from original issue: kubernetes/kubernetes#36987

[luxas]

@dgoodwin I think this issue is fixed with your PR kubernetes/kubernetes#36083, right?

[dgoodwin]

@luxas no I don't think so as this is /var/lib/kubelet, which I didn't touch in that pr. When combined with kubernetes/kubernetes#37063 I think something is up, probably deb specific. It almost seems like kubeadm init is being run by default out of the box after package installation.

@ndtreviv what was actually in /var/lib/kubelet? Did it look like this gist? https://gist.github.com/bronger/92d8cf703628c6d1ff9e93aa920515de

[dpaks]

@luxas The issue still persists. The directory /var/lib/kubelet contains two empty directories pods and plugins.

I faced the above issue when I executed the command

kubeadm init

just after installation of the kube* packages.

[vramakrishnan]

I am facing the same issue on Slave node, with the same set of steps.

root@kube-node-3:~# kubeadm join --token=<...> 10.2.15.7
[kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters.
[preflight] Running pre-flight checks
[preflight] Some fatal errors occurred:
/var/lib/kubelet is not empty
[preflight] If you know what you are doing, you can skip pre-flight checks with --skip-preflight-checks

root@kube-node-3:~# cat /home/kubeadmin/install.sh
#!/bin/bash

apt-get update && apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat < /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update

Install docker if you don't have it already.

apt-get install -y docker.io
apt-get install -y kubelet kubeadm kubectl kubernetes-cni

root@kube-node-3:~# history
1 /home/kubeadmin/install.sh
2 exi
3 exit
4 kubeadm join --token=d7da3b.d259ae8726c2573a 10.2.15.7

root@kube-node-3:~# kubeadm version
kubeadm version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.2074+a092d8e0f95f52", GitCommit:"a092d8e0f95f5200f7ae2cba45c75ab42da36537", GitTreeState:"clean", BuildDate:"2016-12-13T17:03:18Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

root@kube-node-3:~# uname -a
Linux kube-node-3 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

-- I tried to skip the pre-flight checks and it seems working, but not sure of if it missing anything.

root@kube-node-2:~# kubeadm join --token=<..> 10.2.15.7 --skip-preflight-checks
...
...

Node join complete:

• Certificate signing request sent to master and response
  received.
• Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.

[dhague]

As a workaround for this, I run kubeadm reset on the worker node before running kubeadm join ... - this gets rid of the "preflight check" error.