Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubeadm on arm... #18

Closed
mikedanese opened this issue Nov 22, 2016 · 10 comments
Closed

Kubeadm on arm... #18

mikedanese opened this issue Nov 22, 2016 · 10 comments
Assignees

Comments

@mikedanese
Copy link
Member

From @derailed on October 26, 2016 17:23

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

kubeadm arm

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Bug

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.3", 
GitCommit:"4957b090e9a4f6a68b4a40375408fdc74a212260", GitTreeState:"clean", 
BuildDate:"2016-10-16T06:36:33Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/arm"}

Environment:

Raspberry Pi 3

  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):

HypriotOS v 1.0

  • Kernel (e.g. uname -a):

Linux m10 4.4.15-hypriotos-v7+ #1 SMP PREEMPT Mon Jul 25 08:46:52 UTC 2016 armv7l GNU/Linux

  • Install tools:
  • Others:

What happened:

Following kubeadm installation docs. I've installed the prereqs and proceeded with the init as follows:

kubeadm init --use-kubernetes-version v1.4.1 --pod-network-cidr=10.244.0.0/16

The command is stuck on:

master/pki> created keys and certificates in "/etc/kubernetes/pki"
<util/kubeconfig> created "/etc/kubernetes/kubelet.conf"
<util/kubeconfig> created "/etc/kubernetes/admin.conf"
<master/apiclient> created API client configuration
<master/apiclient> created API client, waiting for the control plane to become ready

Looking at docker on this node expecting to see images being installed for etcd, api-server, etc..
but docker images reports nothing. Guessing kubeadm is somehow no able to connect with local docker daemon? or the default Hypriot docker configuration is not jiving with kubeadm??

What you expected to happen:

kubeadm init to complete successfully

How to reproduce it (as minimally and precisely as possible):

o Download Hypriot v1.0 image and flash on SD card
o Boot rp with SD card
o curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
o cat < /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
o apt-get update
o apt-get install -y kubelet kubeadm kubectl kubernetes-cni
o kubeadm init --use-kubernetes-version v1.4.1 --pod-network-cidr=10.244.0.0/16

Anything else do we need to know:

Looking at /etc/kubernetes/manifests shows:

etcd.json kube-apiserver.json kube-controller-manager.json kube-scheduler.json

Inspecting journalctl on kubelet shows:

Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.467776    2787 kubelet_node_status.go:203] Setting node annotation to enable volume controller attach/detach
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.469033    2787 interface.go:248] Default route transits interface "eth0"
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.470261    2787 interface.go:93] Interface eth0 is up
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.471170    2787 interface.go:138] Interface "eth0" has 2 addresses :[192.168.0.92/24 fe80::ba27:ebff:fe24:986f/64].
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.471484    2787 interface.go:105] Checking addr  192.168.0.92/24.
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.471664    2787 interface.go:114] IP found 192.168.0.92
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.471831    2787 interface.go:144] valid IPv4 address for interface "eth0" found as 192.168.0.92.
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.471960    2787 interface.go:254] Choosing IP 192.168.0.92
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.497415    2787 kubelet_node_status.go:354] Recording NodeHasSufficientDisk event message for node m10
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.497616    2787 kubelet_node_status.go:354] Recording NodeHasSufficientMemory event message for node m10
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.497905    2787 kubelet_node_status.go:354] Recording NodeHasNoDiskPressure event message for node m10
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.498077    2787 kubelet_node_status.go:73] Attempting to register node m10
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.498243    2787 server.go:608] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"m10", UID:"m10", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeHasSufficientDisk' Node m10 status is now: NodeHasSufficientDisk
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.498554    2787 server.go:608] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"m10", UID:"m10", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeHasSufficientMemory' Node m10 status is now: NodeHasSufficientMemory
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.498823    2787 server.go:608] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"m10", UID:"m10", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeHasNoDiskPressure' Node m10 status is now: NodeHasNoDiskPressure
Oct 26 17:19:31 m10 kubelet[2787]: E1026 17:19:31.501189    2787 kubelet_node_status.go:97] Unable to register node "m10" with API server: Post https://192.168.0.92:443/api/v1/nodes: dial tcp 192.168.0.92:443: getsockopt: connection refused
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.527656    2787 reflector.go:249] Listing and watching *api.Pod from pkg/kubelet/config/apiserver.go:43
Oct 26 17:19:31 m10 kubelet[2787]: E1026 17:19:31.531088    2787 reflector.go:203] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get https://192.168.0.92:443/api/v1/pods?fieldSelector=spec.nodeName%3Dm10&resourceVersion=0: dial tcp 192.168.0.92:443: getsockopt: connection refused
Oct 26 17:19:32 m10 kubelet[2787]: I1026 17:19:32.416745    2787 reflector.go:249] Listing and watching *api.Service from pkg/kubelet/kubelet.go:384
Oct 26 17:19:32 m10 kubelet[2787]: E1026 17:19:32.418995    2787 reflector.go:203] pkg/kubelet/kubelet.go:384: Failed to list *api.Service: Get https://192.168.0.92:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.92:443: getsockopt: connection refused
Oct 26 17:19:32 m10 kubelet[2787]: I1026 17:19:32.455119    2787 reflector.go:249] Listing and watching *api.Node from pkg/kubelet/kubelet.go:403
Oct 26 17:19:32 m10 kubelet[2787]: E1026 17:19:32.457560    2787 reflector.go:203] pkg/kubelet/kubelet.go:403: Failed to list *api.Node: Get https://192.168.0.92:443/api/v1/nodes?fieldSelector=metadata.name%3Dm10&resourceVersion=0: dial tcp 192.168.0.92:443: getsockopt: connection refused
Oct 26 17:19:32 m10 kubelet[2787]: I1026 17:19:32.531641    2787 reflector.go:249] Listing and watching *api.Pod from pkg/kubelet/config/apiserver.go:43
Oct 26 17:19:32 m10 kubelet[2787]: E1026 17:19:32.534040    2787 reflector.go:203] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get https://192.168.0.92:443/api/v1/pods?fieldSelector=spec.nodeName%3Dm10&resourceVersion=0: dial tcp 192.168.0.92:443: getsockopt: connection refused
Oct 26 17:19:33 m10 kubelet[2787]: I1026 17:19:33.419424    2787 reflector.go:249] Listing and watching *api.Service from pkg/kubelet/kubelet.go:384
Oct 26 17:19:33 m10 kubelet[2787]: E1026 17:19:33.421922    2787 reflector.go:203] pkg/kubelet/kubelet.go:384: Failed to list *api.Service: Get https://192.168.0.92:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.92:443: getsockopt: connection refused

Copied from original issue: kubernetes/kubernetes#35643

@mikedanese
Copy link
Member Author

From @errordeveloper on October 26, 2016 17:28

cc @luxas

@mikedanese
Copy link
Member Author

From @luxas on October 26, 2016 17:31

Did you use Hypriot v1.0.1 as stated in docs?

@mikedanese
Copy link
Member Author

From @nlamirault on October 26, 2016 18:39

@luxas I've got the same issue with Hypriot 1.0.0. Where i can download the 1.0.1 image ?
I try to download https://downloads.hypriot.com/hypriotos-rpi-v1.0.1.img.zip, but i've got an error.
Could you send the link for documentation which explain how use Kubeadm on Hypriot. I could try a new installation.

@mikedanese
Copy link
Member Author

From @derailed on October 26, 2016 18:49

Nicolas - try https://github.com/hypriot/image-builder-rpi/releases

On Wed, Oct 26, 2016 at 12:40 PM, Nicolas Lamirault <
notifications@github.com> wrote:

@luxas https://github.com/luxas I've got the same issue with Hypriot
1.0.0. Where i can download the 1.0.1 image ?
I try to download https://downloads.hypriot.com/
hypriotos-rpi-v1.0.1.img.zip, but i've got an error.
Could you send the link for documentation which explain how use Kubeadm on
Hypriot. I could try a new installation.


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
kubernetes/kubernetes#35643 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAP3KdNzCgFef85JvCFpd-HuE493uidks5q356sgaJpZM4KhcUp
.

@mikedanese
Copy link
Member Author

From @derailed on October 26, 2016 19:25

Thanks for pointing this out @luxas! This was indeed a pilot error. Did get further so we can close this issue. Tho the master is now coming up, setting up the pod network using flannel per the docs results in an error.

ARCH=arm curl -sSL https://raw.githubusercontent.com/luxas/flannel/update-daemonset/Documentation/kube-flannel.yml | sed "s/amd64/${ARCH}/g" | kubectl create -f -

Yields

Error from server: error when creating "flannel.yml": DaemonSet in version "v1beta1" cannot be handled as a DaemonSet: [pos 1115]: json: expect char '"' but got char 'n'

Think the node selector config in the flannel config is incorrect

nodeSelector:
beta.kubernetes.io/arch:

@mikedanese
Copy link
Member Author

From @derailed on October 26, 2016 19:31

Probably a bug in the template, me think the nodeSelector should be:

nodeSelector:
beta.kubernetes.io/arch: arm

@mikedanese
Copy link
Member Author

From @luxas on October 26, 2016 19:34

Ok, seems like the ARCH=arm thing is working poorly then.

Try just running

curl -sSL https://raw.githubusercontent.com/luxas/flannel/update-daemonset/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

@mikedanese
Copy link
Member Author

From @derailed on October 26, 2016 21:26

Thank you all for the prompt support!
Finally got this thing up and running. Watching this cluster live on rpi is a beautiful thing. Totally impressed by your work. kubeadm rocks. Tx!!

@mikedanese
Copy link
Member Author

From @nlamirault on October 27, 2016 11:52

using v1.4.4, process go to the end :

$ sudo kubeadm init --use-kubernetes-version v1.4.4 --api-advertise-addresses=192.168.1.23 --pod-network-cidr=10.244.0.0/16
<master/tokens> generated token: "482d60.042c2504ce81cd32"
<master/pki> created keys and certificates in "/etc/kubernetes/pki"
<util/kubeconfig> created "/etc/kubernetes/kubelet.conf"
<util/kubeconfig> created "/etc/kubernetes/admin.conf"
<master/apiclient> created API client configuration
<master/apiclient> created API client, waiting for the control plane to become ready
<master/apiclient> all control plane components are healthy after 38.573899 seconds
<master/apiclient> waiting for at least one node to register and become ready
<master/apiclient> first node is ready after 1.535058 seconds
<master/discovery> created essential addon: kube-discovery, waiting for it to become ready
<master/discovery> kube-discovery is ready after 814.040634 seconds
<master/addons> created essential addon: kube-proxy
<master/addons> created essential addon: kube-dns

Kubernetes master initialised successfully!

You can now join any number of machines by running the following on each node:

kubeadm join --token 482d60.042c2504ce81cd32 192.168.1.23

But kubectl fails :

$ kubectl get pods --namespace=kube-system
client: etcd cluster is unavailable or misconfigured

The etcd container logs : https://gist.github.com/nlamirault/d84e77e02276d158493f15f249324ba5

@mikedanese
Copy link
Member Author

From @nlamirault on October 27, 2016 12:14

i try the unstable version of kubeadm with reset command. Then i try another init. Etcd container is up, but i've got logs :

2016-10-27 12:14:14.104739 E | etcdhttp: got unexpected response error (etcdserver: request timed out)
2016-10-27 12:14:23.319372 E | etcdhttp: got unexpected response error (etcdserver: request timed out)
2016-10-27 12:14:34.064923 E | etcdhttp: got unexpected response error (etcdserver: request timed out)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants