-
Notifications
You must be signed in to change notification settings - Fork 709
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kubeadm upgrade plan not working for v1.13.5 to v1.14.0 #1469
Comments
thanks for the report. |
here is what get, this is a bit of a bug on it's own because it's telling me that i'm up to date while it should be telling me to update to v1.14.0. i will log a bug about this but my etcd manifest looks like this:
did you happen to create this cluster using 1.12 before upgrading to 1.13? i remember that we did some changes to the etcd addresses related to HA setups.
the upgrade worked for me. |
logged: |
@neolit123 Thanks for the comments. The cluster was initially created using kubeadm v1.9.x. Later on rebuilt, definitely used version before v.1.12.0. No wonder the etcd static pod manifest is different. What exactly has changed for I'll try to generate new static pod manifests using latest version from a different machine and see if I can figure it out (also the dependencies). |
@terrywang more details here: that said i think we had a way to handle this type of upgrade transparently between 1.12 and 1.13, so your 1.13 etcd manifest should have been auto-converted to use the network interface address. possibly something went wrong in the process, but also this is the first report we are seeing related to this. please let me know if you remember anything like modifying the etcd manifests manually, which could have broke our 1.12->1.13 logic. |
closing in favor of: #1471 |
@neolit123 Thanks again for the info. Good to know. I've regenerated the static pod manifests using latest version of
However, running root@k8s-node-1:~# kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
error syncing endpoints with etc: context deadline exceeded Will follow #1471 to regenerate the certificates for etcd when I have time. |
Thanks terry for the info, I had the same problem. Also following #1471 My original cluster originated from Kubernetes 1.8, and rebuild during the 1.11 upgrade because broke everything. My etcd also listens to localhost only:
|
+1
|
fix should be up in 1.14.1 |
Update: I waited for kubeadm 1.14.1, it didn't actually fix the issue... Luckily, simply by following the steps in #1471 mentioned by @mauilion I was able to leverage kubeadm phase ( The reason why [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
error syncing endpoints with etc: context deadline exceeded The certificate SAN should look like below X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Alternative Name:
DNS:k8s-node-1, DNS:localhost, IP Address:192.168.100.21, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1, IP Address:10.192.0.2 |
hm, it should have. the PR that @fabriziopandini created was merged and tested by at least a couple of people.
and your existing cert was missing |
Yes, existing etcd server certificate SAN was missing I may have forgotten to restore the static pod manifest for etcd (added node's private IP inside a VPC subnet to Anyway, the problem is well solved. Really appreciate your input and assistance, enjoyed the learning experience ;-) |
yes, that may be the cause. |
Is this a BUG REPORT or FEATURE REQUEST?
Bug Report
Versions
kubeadm version (use
kubeadm version
):v.1.14.0
Environment:
Kubernetes version (use
kubectl version
): v1.13.5Cloud provider or hardware configuration: AWS EC2
OS (e.g. from /etc/os-release): Ubuntu 16.04.6 LTS
Kernel (e.g.
uname -a
): Linux k8s-node-1 4.4.0-143-generic Always enable RBAC so the cluster-info ConfigMap can be exposed #169-Ubuntu SMP Thu Feb 7 07:56:38 UTC 2019 x86_64 x86_64 x86_64 GNU/LinuxOthers: kubeadm provisioned single master k8s cluster (3 nodes), this cluster was created using kubeadm when k8s was at v1.9.0.
What happened?
Use kubeadm to ugrade the cluster, v1.13.4 to v1.13.5 was successful. To v1.14.0 failed becaues kubeadm upgrade plan pre-flight checks trying to connec to etcd using the node's private IP (assigned to NIC eth0) instead of the loopback address etcd is binding to.
Error
As per the kubeadm init workflow, single master k8s cluster etcd pod is created via static pod manifests. By looking at the manifest, etcd binds 127.0.0.1 and is not exposed to external world.
What you expected to happen?
kubeadm upgrade plan should work as expected to output the details, just like v1.13.4 to v.1.3.5.
How to reproduce it (as minimally and precisely as possible)?
Follow the upgrade guide, upgrade any v.1.13.x cluster (created using kubeadm) to v1.14.0.
I've tried to change the bind address but it has so many dependencies that breaks more than it fixes. Also tried to expose the pod as NodePort service, tried using iptables rules to forward traffic destined to the IP address (192.168.100.12 in this case) port 2379 to loopback with no luck.
Is there a way to override the etcd endpoint when running
kubeadm upgrade plan
that'll be the easiest solution.Anything else we need to know?
Hmm...
The text was updated successfully, but these errors were encountered: