-
Notifications
You must be signed in to change notification settings - Fork 706
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kubeadm on AWS cloud provider deadlocks because of insufficent permissions #330
Comments
I have the same problem. Also with CentOS 7 / AWS. Also in my case 1.6.6 works fine. In my case
However, even when I reconfigure I have no clue what is the actual cause and what is the consequence. |
No, it's absolutely not that issue.
We absolutely test things a lot. We have automated CI e2e tests that are green: https://k8s-testgrid.appspot.com/sig-cluster-lifecycle#kubeadm-gce-1.7 I think the cloud provider is automatically detected on the kubelet, ref the I want to remind you that cloud provider integrations are experimental as kubeadm will only fully support the out-of-tree cloud providers. The current in-tree providers might just as well work fine, but we can't promise it will work. This is such a case. The kubelet talks to the AWS API, gets a Node Name that is different from the hostname ( So what can we do? As you see, the flow when bootstrapping a cluster using a cloud provider varies from the normal flow and even a lot between providers (AWS is the only provider that does this AFAIK). Do you want to contribute a fix for #64?
I have no AWS credits, if you want to contribute with AWS results, great and thanks 👍 Thanks @Zilman and @scholzj for the bug report, we appreciate it. |
@GheRivero volunteered to work on this 🎉 (i.e. #64 that will solve this problem) |
Status report: First of two PRs is up: kubernetes/kubernetes#48538 I expect this to be fixed in v1.7.1, so that you can specify the node name yourself with |
I can confirm the same behavior with Ubuntu 16.04 The first patch solve that situation for |
Seeing the same issue as @scholzj on CentOS 7.3 with 1.7.0. Falling back to 1.6.6 for now. |
Second part of the fix: kubernetes/kubernetes#48594 |
Workaround for EC2 Ubuntu |
Fixed with v1.7.1 |
I'm still experiencing this with 1.7.3 during |
@gtaylor You must set |
After reading a handful of these issue threads, I think my issue is that I'm setting a non-default hostname, leading to the indefinite deadlocking. Even if kubelet's If I am understanding correctly, you can't use the node authorizer with a hostname that doesn't match the one returned by the AWS API. So you're always going to end up having to stick with the default AWS I hope I am misunderstanding something, because we strongly prefer to change our master and node hostnames! |
i'm getting this hang: with kubernetes v1.7.5 on Ubuntu 17.04. multiple online tutorials, including the official one here do not work out of the box: so i found this comment at github: after running that script |
I ran into this issue also and got it to work by updating the hostname of my EC2 instance to match that of the EC2 metadata service. On Ubuntu I added the following steps to my master bootscript:
(adding to |
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Versions
kubeadm version:
&version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T22:55:19Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Environment:
kubectl version
):Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.6", GitCommit:"7fa1c1756d8bc963f1a389f4a6937dc71f08ada2", GitTreeState:"clean", BuildDate:"2017-06-16T18:21:54Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}
uname -a
): Linux ip-172-20-0-134.us-west-2.compute.internal 3.10.0-514.21.2.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Tue Jun 20 12:24:47 UTC 2017 x86_64 x86_64 x86_64 GNU/LinuxWhat happened?
What you expected to happen?
It's supposed to continue with:
How to reproduce it (as minimally and precisely as possible)?
That's the only change and this time the init succeeds (which is where the second paste above comes from).
Anything else we need to know?
This seems to be a regression/repeat of kubernetes/kubernetes#43815, which is a bummer.
Kubernetes v1.7.0 got released two days ago and, well, does kubeadm v1.7.0 get built and released in tandem automatically or something? Nobody tests it first?
People who didn't pin to kubeadm v1.6.6 will suddenly get a bunch of broken clusters this week.
The main error to look out for is once again:
Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Reproducibly only happens on v1.7.0.
I really hope we do better and catch such things in time for v1.8.0 and am working on an automated test suite for this, if anybody is interested in that please ping me.
The text was updated successfully, but these errors were encountered: