"kubeadm init" not possible on restricted network

[rschoultz]

This is a...

• Feature Request
• Bug Report

Problem:
Starting in version 1.6.x, it is not possible to run "kubeadm init" from a private network, i.e. where the host is not allowed to access Internet, as kubeadm get stuck on trying to retrieve a stable-1.6.txt file:

# kubeadm init
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
unable to get URL "https://storage.googleapis.com/kubernetesrelease/release/stable-1.6.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.6.txt: dial tcp 216.58.204.80:443: i/o timeout

I have, of course, tried with flag combinations like "kubeadm init --skip-preflight-checks --kubernetes-version=stable-1.6.4", but it seems like kubeadm from some version 1.6.x cannot run without access to the Internet.

Proposed Solution:
Please make the access to external services optional.

This is on using version 1.6.4:

Versions

# kubeadm version
kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version): 1.6.4
• Cloud provider or hardware configuration: Private
• OS (e.g. from /etc/os-release): CentOS 7.2
• Kernel (e.g. uname -a): Linux *** 3.10.0-327.36.3.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

**Moved from where originally registered in kubernetes/website#4019

[luxas]

Hi @rschoultz!

It is possible to deploy kubeadm without internet. You have to prepull the required master images for the version of choice in beforehand though.

Sorry that the --kubernetes-version flag caused you trouble. For convenience, it tries to get the latest stable version from the internet (and we don't have to hard-code such a value, which would be bad), and consequently fails when there is no internet.

However, you might skip the check by passing a valid semver kubernetes version directly like this:

kubeadm init --kubernetes-version v1.6.4

I noticed that you used --skip-preflight-checks above. Can you elaborate why you did that? Was it to try to workaround this or something else?

I'm taking a pass at docs currently, and I will incorporate this feedback to address this. Also we should give a much more user-friendly output of the command in this situation.

Hope this helps and thanks for using kubeadm 😄!

[rschoultz]

OK, yes, I had tried the --kubernetes-version flag as well, I thought, but I did not supply it correctly, as I had indicated "1.6.4" and not "v1.6.4".

The reason for using --skip-preflight-checks was just for trying out various potential options for working around the problem.

[ReSearchITEng]

facing the same issue, in envs with proxy:
kubeadm starting 1.6.4 tries to connect to

"stderr": "unable to get URL \"https://storage.googleapis.com/kubernetes-release/release/stable-1.6.txt\": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.6.txt: dial tcp: lookup storage.googleapis.com on 10.1.2.3:53: no such host"

If I set the proxy, there are 2 issues:

1. it fails later on, because when it's trying to connect to itself, it's using ip address (as opposite to hostname with fqdn), and the NO_PROXY does not catch this exception.

[preflight] WARNING: Connection to "https://10.2.3.5:6443" uses proxy "http://proxy.corp.example.com:8080/". If that is not intended, adjust your proxy settings

In the hope that it won't use IP address, but hostnames, I tried the below, but for some reason it's still working with IP addresses :(, and the NO_PROXY is not able to skip them, as no_proxy does not work with wildcards...

--apiserver-advertise-address", "machine1.corp.example.com,machine1", "--apiserver-cert-extra-sans", "machine1.corp.example.com,machine1,kubernetes", "--service-dns-domain", "k8s.cloud.corp.example.com",

As a workaround, for now I have set NO_PROXY for each IP address individually.

2. Even worst is that, if the proxy env vars are defined when kubeadm is being run, it will automatically populate it under all relevant /etc/kubernetes/manifests/*.yaml

kube-apiserver.yaml:    - name: NO_PROXY
kube-controller-manager.yaml:    - name: NO_PROXY
kube-scheduler.yaml:    - name: NO_PROXY

...
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    env:
    - name: NO_PROXY
...

Note: should anyone plan to use proxy with kubeadm, being a go application, honors only NO_PROXY environment variable, and ignores the no_proxy (lowecase). (see golang/go#13456 )

To replicate this issue with 1.6.4, one can use: https://github.com/ReSearchITEng/kubeadm-playbook

[luxas]

@ReSearchITEng

kubeadm starting 1.6.4 tries to connect to

You can set --kubernetes-version v1.6.4 to skip that fetch from the internet.

cc @kad

[ReSearchITEng]

Digging further, it seems my issue comes from --config=/etc/kubernetes/kubeadm.conf
Meaning, if --config is defined, it will ignore all the command line arguments, not merge them with the defaults in conf file... (https://kubernetes.io/docs/admin/kubeadm/)

[luxas]

@ReSearchITEng --config always takes priority over CLI flags. In v1.7, kubeadm will error out if you try such an invalid set of params. v1.6.x unfortunately doesn't

[ReSearchITEng]

@luxas Taking priority would be ok, but it's completely ignoring the cli arguments.
My situation was: I added in file config only one parameter (which does not exist as cli), and kept the rest as cli.
To my surprise it did not merge them, it simply discarded all the cli arguments.

[luxas]

@ReSearchITEng That's due how to the API machinery in Kubernetes works, I'm sorry. Therefore, we've added better validations for such cases in v1.7

[kad]

@ReSearchITEng btw, golang respects both upper and lowercase *_proxy variables. you can see how it's done here: https://github.com/golang/go/blob/master/src/net/http/transport.go#L266 and https://github.com/golang/go/blob/master/src/net/http/transport.go#L568

Regarding v1.6.4 vs. 1.6.4 indeed, might be either documentation can be improved or handling of that parameter. I'll look.

[luxas]

Kind of fixed with kubernetes/kubernetes#48507