Run kubeadm init command on the host that already create etcd with kubeadm

[pytimer]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:14:39Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):

Linux master1 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

• Others:

kubeadm configuration file:

apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
etcd:
    external:
        endpoints:
        - https://192.168.100.10:2379
        caFile: /etc/kubernetes/pki/etcd/ca.crt
        certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
        keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
networking:
    podSubnet: 10.204.0.0/16

What happened?

I create Highly Availabile etcd Cluster with kubeadm successfully.

Now i want to init Kubernetes Cluster on the three etcd host, when i run kubeadm init --config kubeadm.yaml, it is failed.

Below error log:

[preflight] Some fatal errors occurred:
        [ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

What you expected to happen?

I found kubeadm code checks.go#L865 always check etcd static pod file, this check should be cancelled if use external etcd cluster?

How to reproduce it (as minimally and precisely as possible)?

• create Highly Availabile etcd Cluster with kubeadm on three host

• kubeadm init --config kubeadm.yaml on the etcd host

Anything else we need to know?

[fabriziopandini]

@chuckha could you kindly take a look at this?

[neolit123]

i think it makes sense:

I found kubeadm code checks.go#L865 always check etcd static pod file, this check should be cancelled if use external etcd cluster?

if the etcd certs were already created and if the config is setup for a remote etcd, we probably shouldn't fail on that file check. then again the remote/local context is kind of misleading here, because the etcd is local to this VM yet external to that particular kubeadm init invocation. 🤔

[pytimer]

i think this check maybe not necessary, because if use external etcd, vm not create etcd static pod.

[chuckha]

@pytimer it sounds like you're trying to do stacked control plane & etcd. We've got special instructions for that: https://kubernetes.io/docs/setup/independent/high-availability/#stacked-control-plane-nodes

[pytimer]

@chuckha Thank for you reply. I see this doc for create HA cluster, but kubeadm init always check etcd static pod file when i use external etcd. Because i run kubeadm init on the host that already install etcd with kubeadm, so preflight check failed.

I want to kubeadm can support use external or local etcd on the same host.

[chuckha]

local etcd was designed to do exactly what you're trying to do. I don't think we are going to support using a colocated etcd with the external configuration. Is there some use case we might be missing that would require the use of external with a colocated control plane / etcd?

[pytimer]

When the small number of virtual machines, install etcd and control plane on the same host. And the other case, there are more vms, install etcd and control plane on the different hosts. I hope that two case, the installation method is consistent, and connection etcd by vip.

Is this idea correct? Or have other soluation to do it.

[fabriziopandini]

@pytimer you can install etcd on the same machines of the control plane, but in that case my personal raccomandation is to install etcd without using kubeadm.

The reason behind this is that you are using kubeadm for two different scopes on the same machine (install etcd external cluster and install kubernetes using the external cluster), but this is basically obscure to kubeadm itself. As a consequence kubeadm will mix/without distinction all the certificates, manifest etc in the same /etc/kubernetes folder.

This can lead to unexpected behaviour like the preflight error (that you can eventually skip) or even more severe problems if you think about what will happen if you run kubeadm reset.
Also kubeadm upgrade might give some troubles...

Does this makes sense to you?

[fabriziopandini]

Fyi we are working to a different solution that doesn't use the concept of external etcd but extends the current local etcd #1123

[pytimer]

@fabianofranz

I create etcd HA cluster with kubeadm, and then init kubernetes HA cluster with kubeadm.

• use kubeadm alpha phase certs xxx to create control plane ca certs.
• use kubeadm alpha phase certs xxx to create control plane certs.
• use kubeadm init command and skip preflight check.

I run kubectl on the one of master hosts.

[root@master1 ~]# kubectl get pod -n kube-system
NAME                                       READY     STATUS    RESTARTS   AGE
coredns-545dfc9547-5f27b                   1/1       Running   0          18h
coredns-545dfc9547-mfv4d                   1/1       Running   0          18h
etcd-master1.cloud                      1/1       Running   2          29m
etcd-master2.cloud                      1/1       Running   1          17h
etcd-master3.cloud                      1/1       Running   0          17h
kube-apiserver-master1.cloud            1/1       Running   1          29m
kube-apiserver-master2.cloud            1/1       Running   0          17h
kube-apiserver-master3.cloud            1/1       Running   0          17h
kube-controller-manager-master1.cloud   1/1       Running   1          29m
kube-controller-manager-master2.cloud   1/1       Running   0          17h
kube-controller-manager-master3.cloud   1/1       Running   0          17h
kube-flannel-ds-8zkkf                      1/1       Running   0          17h
kube-flannel-ds-bh4zl                      1/1       Running   1          17h
kube-flannel-ds-jdgmf                      1/1       Running   1          15h
kube-flannel-ds-qvh5r                      1/1       Running   0          17h
kube-proxy-f4vct                           1/1       Running   0          15h
kube-proxy-s8vgg                           1/1       Running   0          18h
kube-proxy-stbmg                           1/1       Running   1          18h
kube-proxy-xrgfn                           1/1       Running   0          17h
kube-scheduler-master1.cloud            1/1       Running   1          29m
kube-scheduler-master2.cloud            1/1       Running   0          17h
kube-scheduler-master3.cloud            1/1       Running   0          17h

But like you said, when i run kubeadm reset, etcd static pod killed and all certs removed.

Does kubernetes recommended to use local etcd or external etcd?

Fyi we are working to a different solution that doesn't use the concept of external etcd but extends the current local etcd #1123

I read this issue, is it D) we would like to scale etcd be separated from control plane scaling (e.g kubeadm join --etcd) like my case? If use local etcd, it means that etcd numbers must be equals with control plane instances?

If etcd and control plane install together, etcd instances will increase, is there problem with this?

If kubeadm add join --control-plane feature, it is nice. :-)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[timothysc]

Closing due to inactivity and the HA stacked documents have been cross-verified by multiple parties.