Starting Kubernetes on an Arch Linux VM using Kubeadm

[imriss]

Is this a BUG REPORT or FEATURE REQUEST?

It could be both BUG REPORT and FEATURE REQUEST.

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5+17d7182a7ccbb", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"not a git tree", BuildDate:"2017-09-12T15:19:56Z", GoVersion:"go1.9", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5+17d7182a7ccbb", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"not a git tree", BuildDate:"2017-09-12T15:19:56Z", GoVersion:"go1.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c", GitTreeState:"clean", BuildDate:"2017-09-14T06:36:08Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
ID_LIKE=archlinux
ANSI_COLOR="0;36"
HOME_URL="https://www.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"

• Kernel (e.g. uname -a):
  Linux Arch1 4.12.13-1-ARCH #1 SMP PREEMPT Fri Sep 15 06:36:43 UTC 2017 x86_64 GNU/Linux

What happened?

1. Hanging for a long time (indefinite) at: [apiclient] Created API client, waiting for the control plane to become ready
2. Hanging for a long time (indefinite) at: [apiclient] All control plane components are healthy after 16.56457 seconds

How to reproduce it (as minimally and precisely as possible)?

The [excerpts from the] working configuration:
Note: The VM has two interfaces. 10.0.3.15 is the IP on the NAT interface, while 10.13.13.103 is the IP of the Internal interface.

~/.bashrc:

export GOPATH=/home/user/gocode/
export KUBECONFIG=$HOME/admin.conf

/etc/hosts:

127.0.0.1	localhost.localdomain	localhost 
::1		localhost.localdomain	localhost 
10.0.3.15 	localhost.localdomain	arch1

/etc/kubernetes/config:

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://10.0.3.15:8080"

/etc/kubernetes/kubelet:

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=0.0.0.0"

# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override="

# location of the api-server
KUBELET_API_SERVER="" # --api-servers=http://10.0.3.15:8080"

# Add your own!
KUBELET_ARGS="--pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd --require-kubeconfig --kubeconfig=/home/user/admin.conf"

/usr/lib/systemd/system/docker.service:
ExecStart=/usr/bin/dockerd -H fd:// --exec-opt native.cgroupdriver=systemd

Steps:

Step 0:
systemctl stop kubelet.service
Remove k8s containers, if any.
mount | grep "/var/lib/kubelet/*" | awk '{print $3}' | sudo xargs umount 1>/dev/null 2>/dev/null

Step 1: Change the files and reboot, if needed.

Step 2:

systemctl daemon-reload
systemctl restart docker.service
systemctl status docker.service
systemctl restart kubelet.service
systemctl status kubelet.service

Step 3:
In a terminal, switch to root, then:

kubeadm reset
kubeadm init

Then, wait until you see the message "All control plane components are healthy after 16.56457 seconds".

Step 4: In another terminal:

sudo cp /etc/kubernetes/admin.conf $HOME/
sudo chown $(id -u):$(id -g) $HOME/admin.conf
export KUBECONFIG=$HOME/admin.conf

Then:
systemctl restart kubelet.service

Step 5:
You will see the progress in the first terminal (where you issued kubeadm init) to the end where you receive the token:
kubeadm join --token ecb54a.3456765423dd34c2 10.0.3.15:6443

What is expected?

It is expected that kubeadm init performs a systemctl restart kubelet.service at some point because it knows that the credentials have been changed.

Anything else we need to know?

1. Hanging for a long time (indefinite) at: [apiclient] Created API client, waiting for the control plane to become ready was mentioned:
   kubeadm 1.7.2 init blocks on waiting for control plane to be ready #360
   Create a master node takes a long time. kubernetes-retired/kubeadm-dind-cluster#34
   kubeadm 1.7.2 init blocks on waiting for control plane to be ready #360 (comment)
   kubeadm init waiting for the control plane to become ready on CentOS 7.2 with kubeadm 1.6.1 #228
   kubeadm init hang up at 'Created API client, waiting for the control plane to become ready', but I'm not using systemd or selinux #353
   kubeadm init hangs CentOS 7 kubeadm 1.6 #308
   kubeadm blocks waiting for 'control plane' kubernetes#33544

2. Hanging for a long time (indefinite) at: [apiclient] All control plane components are healthy after 16.56457 seconds was mentioned:
   kubelet can not up of V1.5.0  kubernetes#39602
   Running Kubernetes Locally via Docker - kubectl get nodes returns The connection to the server localhost:8080 was refused - did you specify the right host or port? kubernetes#23726
   1.6.1 The connection to the server localhost:8080 was refused kubernetes#44665

3. Avoiding --api-servers was mentioned:
   kubelet: Incorrect fallback to hairpin-mode "hairpin-veth" kubernetes#23322
   Kubelet won't read apiserver from kubeconfig kubernetes#36745

[jamiehannaford]

@imriss I'm not exactly sure what the issue is here. Is it that you expect kubeadm to restart kubelet?

[imriss]

@jamiehannaford Yes, or alternately send a notification and ask the user to manually restart kubelet.

[jamiehannaford]

@imriss Why do you need to restart kubelet on a fresh VM that kubeadm has just been installed to?

Or is this for VMs that had previous clusters on? Doesn't kubeadm reset followed by systemctl restart kubelet work?

[kad]

How are other components are installed ? kubeadm expects to have specific systemd service file to be used for kubelet, and if you're trying to mix components packaged in ArchLinux that have different settings, it might not work as expected.

[imriss]

@jamiehannaford @kad Thanks for the insights.
The main step in my case is that kubelet needs to use the new credentials that kubeadm generates during the init process. I need to restart kubelet.service for this. Otherwise, kuebadm init has no progress after [apiclient] All control plane components are healthy after ... message.
Thanks.

[imriss]

@kad For installation, I am using aur kubernetes package without any modifications.

[jamiehannaford]

@imriss Yeah, so as @kad already mentioned, I think there's some inconsistency with the installation expectations across distros. When kubeadm is first installed with apt, for example, it creates a systemd drop-in that set's kubelet's --kubeconfig. When the file is eventually written, kubelet should dynamically pick this up without the need for a restart. However, for other Linux types that don't have this, I imagine you'll need to manually reconfigure things. We should probably document that.

/cc @luxas

[imriss]

@jamiehannaford Thanks. I set the --kubeconfig to --kubeconfig=/home/user/admin.conf in the /etc/kubernetes/kubelet file. I manually copy that file from /etc/kubernetes/admin.conf to $HOME. Is there any specific time interval that kubelet looks for any change in the --kubeconfig file? I will also try to set the --kubeconfig to /etc/kubernetes/admin.conf, and see if it picks it up.

[vkumar2808]

Hello,

Did it work? im having same issue

[imriss]

@vkumar2808 No, it still needs kubelet restart the way I listed above with some changes.

1. The changes to nano /etc/kubernetes/kubelet (relative to what is listed above):

KUBELET_ARGS="--pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd --fail-swap-on=false --require-kubeconfig --kubeconfig=/etc/kubernetes/admin.conf"

2. The changes to Step 2:
kubelet service would not start (there is no /etc/kubernetes/admin.conf yet). However, it is fine.

3. The changes to Step 3:
Then, wait until you see the message "[kubeconfig] Wrote KubeConfig file to disk: "admin.conf""

4. The NEW Step 4:
In another terminal:
systemctl restart kubelet.service

[vkumar2808]

Thanks @imriss . Could you please send me sample of your kublet conf file?

[imriss]

@vkumar2808 This is the content of /etc/kubernetes/config file:

###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://10.0.3.15:8080"

[imriss]

For the recommended $HOME/.kube/config place to put admin.conf, the procedure is the same. Except, setting the $KUBECONFIG would be optional.

[vkumar2808]

Thanks @imriss . I just deployed k8s v 1.8.1 on single server and it is seamlessly working fine. I had several sleepless night and weekend to work on it. I can seriously say that kubernetes document is lacking some important stuff. It may be because they have widen deployment for several platform.
Now Im very careful about each and every word written in kubernetes document. it has implied meaning and relation somewhere.
I tried with several plugin for centos 7 and calico worked fine for me. My main trouble was kubelet was not starting after server reboot but it is now. I will suggest people to use this link and in 15 min you can setup your local cluster.

https://docs.projectcalico.org/v2.6/getting-started/kubernetes/

My QA needs dashboard to access logs and it is great now..everything has started fine.

My next adventure will be install k8s on multi nodes and make sure after server reboot it survive. i have written several command and notes but nothing can beat calico instruction.

[luxas]

Are there action items for kubeadm here? There are no official packages for Arch Linux for kubeadm AFAIK. cc @colemickens
You basically need to install the contents of the debs (converted to tar files for instance), and add this content to your rootfs and it should work.

[colemickens]

There's no official package, but I do maintain kubeadm-bin (pacaur -S kubeadm-bin or yaourt -S kubeadm-bin, etc). It, with a couple other packages that should be pulled in transitively, along with an actual kubeadm init should be about all that's needed. I would expect multi-node to be a fairly straightforward extension.

I keep some scripts around to help with destroying/recreating my singlen node cluster here (https://github.com/colemickens/chimera-k8s-config), but you'll see the cluster boot is largely just:

1. Install kubeadm/kubelet.
2. kubeadm reset
3. kubeadm init
4. install calico

(One other small note, Arch packages don't start/enable services by default, hence why I don't have the kubelet or kubeadm packages enable/start kubelet.

[imriss]

Thanks all. As @colemickens mentioned there are many packages that provide kubernetes on Arch Linux. I have all the binaries installed. The purpose of this issue filing was to have a documented list of steps to bring up k8s, such as cgroup/systemd setting, among others. From all the comments I am learning more. Thanks again.

[vkumar2808]

Hey Guys, Use flannel. With calico you may run into issue for several setup for multiple nodes. Flannel works for me for multi node without much work for setup.

…

On Sat, Oct 21, 2017 at 6:33 PM, Reza Farrahi M ***@***.***> wrote: Thanks all. As @colemickens <https://github.com/colemickens> mentioned there are many packages that provide kubernetes on Arch Linux. I have all the binaries installed. The purpose of this issue filing was to have a documented list of steps to bring up k8s, such as cgroup/systemd setting, among others. From all the comments I am learning more. Thanks again. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#465 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATbohDW7-czK6JlJADGHy1bjqXi-1TDnks5sunEngaJpZM4PiyrQ> .

-- Thanks Vijay

[colemickens]

@imriss What other steps did you need to do? At one point I thought I'd had to tweak the cgroup driver, but I'm not setting anything in the kubeadm config, and the only docker config I'm providing is changing the storage driver. Right now, as far as I can tell, I'm blowing away all Kubernetes config on my system and re-deploying it with every release of Kubernetes and all of those steps are captured in the script(s) linked above.

@vkumar2808 can you provide any more details? Why did Calico not work in a multi-node setup? What didn't work?

[imriss]

@colemickens sorry for the delay. One other thing is the need to offline-download the images (mentioned here and here) before kubeadm init.

[imriss]

The command to pull the images (to have it offline before calling kubeadm init):
echo "gcr.io/google_containers/etcd-amd64:3.0.17 quay.io/coreos/flannel:v0.8.0-amd64 gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5 gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5 gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5 gcr.io/google_containers/kube-apiserver-amd64:v1.8.0 gcr.io/google_containers/kube-controller-manager-amd64:v1.8.0 gcr.io/google_containers/kube-proxy-amd64:v1.8.0 gcr.io/google_containers/kube-scheduler-amd64:v1.8.0 gcr.io/google_containers/pause-amd64:3.0 " | tr " " "\n" | xargs -L 1 sudo docker pull

[colemickens]

I don't understand why those are necessary. kubeadm just instructs docker to run things which will pull on demand. Or was this specific to your use-case where you need to actually run the cluster offline after initially pulling down the container assets?

[errordeveloper]

@imriss I cannot see a specific issue here that we need to fix. The information you have provided appears to document an installation process on Arch Linux. If you are calling for improvements of user experience on Arch Linux, please let us know and we will see what can be done. If simply would like to document the process, please write a blog post or consider opening a PR to add it to the docs (if it's something you are happy to maintain).

[timothysc]

Closing, stale and last comment.