-
Notifications
You must be signed in to change notification settings - Fork 706
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kubelet won't restart after reboot - Unable to register node with API server: connection refused #1026
Comments
hello @PierrickI3 do you have connectivity to:
|
I don't, but it's my local IP address:
|
6443 is the secure port of the API server. what is the output of |
i haven't seen a failure right after do you happen to have a firewall blocking 6443? |
Here is the output from kubectl:
I don't have any firewall blocking this port and I ssh'd directly on the machine. Here is the manifest (no sensitive data to hide as it's not exposed):
|
here is the output from docker ps:
|
this one should be deprecated and disabled by default in 1.11.1, did you adapt an old kubeadm config? what are the contents of your kubeadm config (again, please hide sensitive data where needed)? |
How can I retrieve the kubeadm config? Running To deploy, I followed the kubeadm instructions. Here is what I executed:
|
in this case you are not passing what happens if you use a different CNI? first this without CIDR:
then:
edit: play with removing the taint line: |
should I run |
yes, unless you have are reason to not want to? |
I ran
|
Here is the
|
i've been seeing this the past few days from users, for some reason.... can you pre-pull the images using:
then try |
here is the output from
Then ran
Then
journalctl last entries:
I am not able to add any network pod yet. |
yes, this is failing earlier.
please, restart the kubelet manually and see what the logs show: systemctl restart kubelet
systemctl status kubelet # <---- ?
journalctl -xeu kubelet # <---- ? i'm running out of ideas. |
journalctl -xeu kubelet | less:
|
what happens if you stop everything related to kubernetes on this node and try to run a small server listening on
write this to a test.go file package main
import (
"net/http"
"strings"
)
func sayHello(w http.ResponseWriter, r *http.Request) {
message := r.URL.Path
message = strings.TrimPrefix(message, "/")
message = "Hello " + message
w.Write([]byte(message))
}
func main() {
http.HandleFunc("/", sayHello)
if err := http.ListenAndServe("192.168.1.19:6443", nil); err != nil {
panic(err)
}
}
i know this is silly, but i'm out of ideas. |
Output of
output of
|
well it works! maybe someone else can have a look at the logs and figure out whats going on? |
Anyone in particular you think of? Could you tag them in this issue? |
@kubernetes/sig-cluster-lifecycle-bugs |
@PierrickI3 can you check if app server is listening on port 6443? If it is, can you check your proxy settings (if any) ?
|
@bart0sh Thanks but I had to start over this weekend so I can no longer troubleshoot this. I will close this issue. FYI, the output of netstat -tulpn is shown above (#1026 (comment)). I have no proxy (direct connection to the internet). |
OMG THANKS!!! |
systemctl status kubelet cmd show the error info So vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf |
This fixed my issue, although i only did this in the master node and just did kubeadm reset in other nodes |
Thanks @manoj-bandara but I gave up for now and will return to this later on this year. See #1026 (comment) |
same problem here - tried a lot of things . it seems this was the problem: i symlinked /etc/resolv.conf to that location and restarted kubele (only on master) then the master and all nodes started to came up again. |
In my case it was the swap issue, fixed it by turning off the swap
|
OMG,amazing command!!Thank u very much indeed!! |
@JTRNEO , I have to remember this one as well :) |
I faced with the same problem several installations. I haven't figured it out yet. I tried Flannel, Calico and Wave drivers and the result is the same. So probably the problem is not related with network plugins. |
Same issue here. My kube-apiserver logs are filled with:
The flushing ip tables didn't work for me and my swap is already off. |
Did you install flannel to the worker nodes?
…On Sun, Nov 24, 2019, 21:54 Christopher J. Bottaro ***@***.***> wrote:
Same issue here. My kube-apiserver logs are filled with:
I1124 18:44:13.180672 1 log.go:172] http: TLS handshake error from 192.168.1.235:56160: EOF
I1124 18:44:13.186601 1 log.go:172] http: TLS handshake error from 192.168.1.235:56244: EOF
I1124 18:44:13.201880 1 log.go:172] http: TLS handshake error from 192.168.1.192:56340: EOF
I1124 18:44:13.208991 1 log.go:172] http: TLS handshake error from 192.168.1.235:56234: EOF
I1124 18:44:13.248214 1 log.go:172] http: TLS handshake error from 192.168.1.235:56166: EOF
I1124 18:44:13.292943 1 log.go:172] http: TLS handshake error from 192.168.1.235:56272: EOF
I1124 18:44:13.332362 1 log.go:172] http: TLS handshake error from 192.168.1.235:56150: EOF
I1124 18:44:13.352911 1 log.go:172] http: TLS handshake error from 192.168.1.249:41300: EOF
The flushing ip tables didn't work for me and my swap is already off.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1026?email_source=notifications&email_token=AKZB2S2OOQWTN5QH3NCIVNTQVLEYBA5CNFSM4FMNY3N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFASFSY#issuecomment-557916875>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKZB2SZZTPQEVNMV77HZFSLQVLEYBANCNFSM4FMNY3NQ>
.
|
No, I'm using kube-router for networking. The pod for it is in crash backoff state, and filled with errors about not being able to talk to the api server. The cluster was running fine for months, but then the electricity went out hence rebooting my machines and a borked cluster. |
@cjbottaro see the second warning here:
|
I am running cluster in vmware too, what did resolve your problem? Thanks |
Had the same issues today after editing the service-cidr settings on my new kube cluster. The issue for me was the kube-apiserver docker container was flapping. After looking at the logs using docker logs <container_id>, I saw: |
not sure if related but note that 1.17.0 has CIDR related bugs, so hopefully you are running a more recent patch of .17. |
u may need to change the rules of iptables. |
I have a simmilar promblem using kubeadm cluster. I've just use 2x times docker restart $(docker ps -qa) |
Have the same problem that @PierrickI3 does. After reboot the node control plane is down. There is no particular error seen anywhere, it just does not start, though worked fine for a long time before the reboot. Tried everything mention in this thread as well as in many others. I am utterly confused on what starts what here to investigate the issue further. Does |
Had similar issue, its not resolved for me yet but the issue occurred due to docker getting upgraded to an incompatible version. Just check of your docker service is running or not. |
Thanks.My machine has been unable to connect after restarting. After checking the certificate, it was found that 3 certificates had expired.
So renew the certificate
Restart service
It works fine |
Hi, |
In my case, kubelet could not find the node because, |
This works ! Had the problem after kubespray and was able to start node1 again with |
I have the similar issue, My set up have 2 master nodes 2 worker nodes and HA proxy. eviction_manager.go:254] "Eviction manager: failed to get summary stats" err="failed to get node info: node "cn-manager1" not found" HA Proxy service is up and running #--------------------------------------------------------------------- apiserver frontend which proxys to the control plane nodes#--------------------------------------------------------------------- #--------------------------------------------------------------------- round robin balancing for apiserver#--------------------------------------------------------------------- |
Hi, |
have same issue - kube-apiserver container is not stable and is exited due to following error -
after checking etcd container it is getting terminated due to following error -
apreciate help on this ; ta |
this is resolved now after I renewed my certificates for etcd which got expired. |
Is this a request for help?
It is but I have searched StackOverflow and googled many times without finding the issue. Also, this seems to affect more people.
What keywords did you search in kubeadm issues before filing this one?
The error messages I see in journalctl
Is this a BUG REPORT or FEATURE REQUEST?
Bug report
Versions
kubeadm version:
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:50:16Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Environment:
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
The server is also 1.11 but since it's not starting at the moment,
kubectl version
won't show itLinux kubernetes 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
What happened?
Kubelet service does not start
What you expected to happen?
Kubelet service should start
How to reproduce it (as minimally and precisely as possible)?
Anything else we need to know?
Journalctl logs:
And it goes on and one about not being able to register
kubernetes
(that's my host name) and failing to list kubernetes resources.From the start, I applied the self-hosted-recover script (https://github.com/xetys/k8s-self-hosted-recovery) to not be affected by a reboot. Here are the logs:
I am running out of ideas and would welcome any help you can bring.
The text was updated successfully, but these errors were encountered: