Kubeadm init fails with "Error writing Crisocket information for the control-plane node: timed out waiting for the condition" #1587
Comments
neolit123
added
priority/awaiting-more-evidence
|
do you see anything suspicions in the kubelet logs? |
|
`~> sudo systemctl status kubelet Jun 02 06:38:50 master-2 hyperkube[14593]: I0602 06:38:50.049825 14593 kubelet_node_status.go:446] Recording NodeHasNoDiskPressure event message for node 127.0.0.1 Kubelet logs: link |
|
Got the same issue: |
|
there are multiple aspects at play here, but i don't know the exact cause.
why not use
hyperkube is not something we have e2e test for, so i wouldn't say the kubeadm team supports it. please try newer versions of k8s and re-open this ticket if the problem persists or if you have found the cause. our e2e test signal for 1.13 is green using containerd and docker. |
|
Kubernetes version Kubeadm version Kubectl get pods: kubectl get nodes -o wide We have been trying to add an additional fourth node to the cluster using kubeadm join command Kubelet service We have seen the error |
could this be a case where the bootstrap token used for join is expired? |
|
@neolist123 we have tried recreating the token so it should still be valid |
|
try enabling --v=10 for "join" and observe the API call failures. |
|
Here's the output of |
|
the --v=10 logs just confirm that it's retrying. what is different about this node? |
|
@neolit123 - Here is the output. I appreciate your help. We have searched all possible google suggestions but unable to resolve this. This started when we did |
|
Ok - Finally, this is solved. The issue was this. When we did And we assumed that it did what it said. When I checked It consumed our whole day. When |
what is your kubeadm version? ultimately kubeadm reset is a best effort command and if something is protected we don't want to touch it. what we can fix here is giving a better indication if a delete failed. |
Thank you as that will be very helpful to know that something did not go right. |
|
i should have asked for the output from we did some refactoring for reset in 1.16 and it's not clear to me whether this is already fixed or not. |
|
I was able to scroll up and get the output for After above: |
|
i will log an issue with your details and investigate if this is fixed in 1.16 |
|
Hi Please the issue is resolved??? [kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster error execution phase upload-config/kubelet: Error writing Crisocket information for the control-plane node: timed out waiting for the condition |
|
for questions please use #kubeadm and the support channels: |
I am not asking a question i have issue with kubeadm init ...can you check the error for how to resolve it ? |
|
the links i gave are the place to ask for support too. you seem to be ignoring the CPU and Memory checks, so probably the machine doesn't have enough memory and the apiserver cannot start properly. |
|
sudo kubeadm reset |
gooooooooood |
|
This fixed for me. #1438 (comment) |
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Versions
kubeadm version (use
kubeadm version): sudo kubeadm versionkubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-29T12:00:00Z", GoVersion:"go1.11.10", Compiler:"gc", Platform:"linux/amd64"}
Environment:
kubectl version):~> kubectl versionClient Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6", GitCommit:"abdda3f9fefa29172298a2e42f5102e777a8ec25", GitTreeState:"clean", BuildDate:"2019-05-08T13:53:53Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6", GitCommit:"abdda3f9fefa29172298a2e42f5102e777a8ec25", GitTreeState:"clean", BuildDate:"2019-05-08T13:46:28Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
cat /etc/os-release
NAME="SLES"
VERSION="15"
VERSION_ID="15"
PRETTY_NAME="SUSE Linux Enterprise Server 15"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15"
uname -a):Linux master-2 4.12.14-150.17-default kubeadm join on slave node fails preflight checks #1 SMP Thu May 2 15:15:46 UTC 2019 (bf13fb8) x86_64 x86_64 x86_64 GNU/LinuxWhat happened?
kubeadm failed with error "Error writing Crisocket information for the control-plane node: timed out waiting for the condition"
What you expected to happen?
"sudo kubeadm init --pod-network-cidr 10.248.0.0/16" command should have setup all the component of master succesfully.
How to reproduce it (as minimally and precisely as possible)?
crayadm@master-2:~> sudo kubeadm init --pod-network-cidr 10.248.0.0/16
I0531 18:16:32.726064 5686 version.go:237] remote version is much newer: v1.14.2; falling back to: stable-1.13
[init] Using Kubernetes version: v1.13.6
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master-2 localhost] and IPs [10.248.0.210 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master-2 localhost] and IPs [10.248.0.210 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master-2 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.248.0.210]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 14.003255 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "master-2" as an annotation
[kubelet-check] Initial timeout of 40s passed.
error execution phase upload-config/kubelet: Error writing Crisocket information for the control-plane node: timed out waiting for the condition.
Anything else we need to know?
~> cat /etc/crictl.yaml
runtime-endpoint: unix:///var/run/dockershim.sock
image-endpoint: unix:///var/run/dockershim.sock
timeout: 10
debug: true
~> crictl pods
FATA[0010] failed to connect: failed to connect: context deadline exceeded
~> systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2019-05-25 14:29:49 UTC; 6 days ago
Docs: http://docs.docker.com
Main PID: 19508 (dockerd)
Tasks: 120
Memory: 103.3M
CPU: 48min 59.396s
CGroup: /system.slice/docker.service
├─ 5878 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/50ba657c71cfeccf8ffd3544334ab2fa9f0576>
├─ 5880 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/830a32f38fec61cd909a31543bc65d2b9b6abe>
├─ 5883 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/95bd179c9c1c90c0f3f97c8e8ee22203b75ecc>
├─ 5897 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/3ccce84564cc38018df299ca780f0929b63fa0>
├─ 5934 /pause
├─ 5941 /pause
├─ 5946 /pause
├─ 5968 /pause
├─ 6045 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/9d66fe3c91d7154e3115620d2c6bc4334548a8>
├─ 6059 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/41fca5033552e43d50a53433228a5f7c3e413c>
├─ 6060 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/a39bffdf8469140338eaabc2d2b490aeaf013b>
├─ 6061 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/2de48a98c6d58d3e7b9322939bd542a9e6a273>
├─ 6094 kube-apiserver --authorization-mode=Node,RBAC --advertise-address=10.248.0.210 --allow-privileged=true --client-ca-file=/etc/kubernetes/pki/ca.crt --enable->
├─ 6111 etcd --advertise-client-urls=https://10.248.0.210:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --in>
├─ 6130 kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-k>
├─ 6140 kube-scheduler --address=127.0.0.1 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true
├─19508 /usr/bin/dockerd --add-runtime oci=/usr/sbin/docker-runc
└─19515 docker-containerd --config /var/run/docker/containerd/containerd.toml
The text was updated successfully, but these errors were encountered: