kubeadm init hangs CentOS 7 kubeadm 1.6

[asatsi]

What keywords did you search in kubeadm issues before filing this one?

kubeadm centos hang

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

This was referenced here to be fixed and to create a new bug if its not fixed. #39
Details of my environment:
$ uname -a
Linux stevevai 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)
$ kubeadm version

Versions

kubeadm version (use kubeadm version):
kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.5", GitCommit:"490c6f13df1cb6612e0993c4c14f2ff90f8cdbf3", GitTreeState:"clean", BuildDate:"2017-06-14T20:03:38Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  $ kubectl version
  Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.5", GitCommit:"490c6f13df1cb6612e0993c4c14f2ff90f8cdbf3", GitTreeState:"clean", BuildDate:"2017-06-14T20:15:53Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  N/A VirtualBox VM with 2GB RAM

• OS (e.g. from /etc/os-release):
  NAME="CentOS Linux"
  VERSION="7 (Core)"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="7"
  PRETTY_NAME="CentOS Linux 7 (Core)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:7"
  HOME_URL="https://www.centos.org/"
  BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):
  Linux stevevai 3.10.0-514.16.1.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Wed Apr 12 15:04:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Others:

docker version

Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:05:44 2017
OS/Arch: linux/amd64

Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:05:44 2017
OS/Arch: linux/amd64
Experimental: false

What happened?

kubeadm init hangs

What you expected to happen?

The command kubeadm init should proceed and complete successfully resulting initialization and startup of kubernetes.

How to reproduce it (as minimally and precisely as possible)?

Use the software versions mentioned above.

Anything else we need to know?

Output of the command:
]# kubeadm init --skip-preflight-checks
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.6.5
[init] Using Authorization mode: RBAC
[preflight] Skipping pre-flight checks
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [stevevai kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.2.15]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready

[pipejakob]

Were the preflight checks not passing for you? What happens if you try kubeadm init without skipping them?

[luxas]

@asatsi Is it possible that you could use docker 1.12 that is the supported version?

[fabriziopandini]

I managed to reproduce the error, and it seems that kubeadm init hangs with docker 17.03.1-ce, while it works with docker 1.12

I also opened #312 to track the problem related to the preflight checks not passing

[mogaal]

I got the same problem and it was because I was using the kernel vmlinuz-4.11.2-1.el7.elrepo.x86_64. I changed to vmlinuz-3.10.0-514.16.1.el7.x86_64 and it was sorted.

[luxas]

@mogaal Interesting. How did the kubelet react/behave on 4.11?

[drazul]

@fabriziopandini that is because the default cgroup driver on docker 1.17 is cgroupfs. You need to change to systemd to work:

You can see the problen executing journalctl -xe and search for kubelet or executing systemctl status kubelet --lines 100

cat << EOF > /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
systemctl restart docker

[imriss]

on Arch Linux, to apply the @drazul solution, you may add --exec-opt native.cgroupdriver=systemd to the end of the line of ExecStart=/usr/bin/dockerd in the /usr/lib/systemd/system/docker.service.

[luxas]

Closing this as solved.