kubeadm on CentOS mounts incompatible /etc/ssl/certs into controller

[mikedanese]

From @codablock on November 3, 2016 10:52

What keywords did you search in Kubernetes issues before filing this one?
controller /etc/ssl/certs
kubeadm /etc/ssl/certs

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):
1.5 custom built master
I also use a custom built hyperkube image instead of the single app images for apiserver, controller, ...

Environment:

• Cloud provider or hardware configuration: azure
• OS (e.g. from /etc/os-release): CentOS 7.2
• Kernel (e.g. uname -a): Linux ma-kub8ms0 3.10.0-327.36.3.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: kubeadm
• Others: deployed with Azure RM template and ansible

What happened:
controller is not able to do https, including calls to the Azure RM API

What you expected to happen:
https should work as usual

How to reproduce it (as minimally and precisely as possible):
Use kubeadm to install kubernetes on CentOS
exec into the controller and try "curl https://google.com"

Anything else do we need to know:
It looks like kubernetes/kubernetes#33681 caused this issue. On CentOS, I see these files on the host OS:

[devops@ma-kub8ms0 ~]$ ls -lah /etc/ssl/certs/
total 12K
drwxr-xr-x. 2 root root  112 Oct 26 20:48 .
drwxr-xr-x. 5 root root   76 Oct 26 20:47 ..
lrwxrwxrwx. 1 root root   49 Oct 26 20:47 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx. 1 root root   55 Oct 26 20:47 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
-rwxr-xr-x. 1 root root  610 Sep 27 13:40 make-dummy-cert
-rw-r--r--. 1 root root 2.4K Sep 27 13:40 Makefile
-rwxr-xr-x. 1 root root  829 Sep 27 13:40 renew-dummy-cert

I suspect the symlinks to /etc/pki/... being the problem here, as they are not available inside the hyperkube Debian image.

I also tried to verify this with gcr.io/google_containers/kube-controller-manager-amd64:v1.4.4. It seems to not be based on Debian and thus gives a different error:

[devops@ma-kub8ms0 ~]$ docker run -ti --rm -v /etc/ssl/certs:/etc/ssl/certs gcr.io/google_containers/kube-controller-manager-amd64:v1.4.4 sh
/ # wget https://google.de
Connecting to google.de (172.217.21.227:443)
wget: can't execute 'ssl_helper': No such file or directory
wget: error getting response: Connection reset by peer

I see 2 possible solutions:

1. Instead of host mounting /etc/ssl/certs into the controller pod, the ca certificates could be installed into the image. This is already done for hyperkube and would work out of the box when the host mount would be omitted.
2. kubeadm could try to figure out if /etc/pki should also be host mounted and add this to the controller manifest if required.

I'd be happy to provide a fix by myself, I just need to know which solution you would prefer.

Copied from original issue: kubernetes/kubernetes#36150

[mikedanese]

From @codablock on November 4, 2016 9:22

Same problem seems to apply to the apiserver. Not sure however if it actually does https calls.

[mikedanese]

From @codablock on November 7, 2016 16:8

kubernetes/kubernetes#36373 implements solution 2.

[codablock]

@luxas @mikedanese Can be closed as kubernetes/kubernetes#36373 was merged.

I can't do this by myself as I'm not the owner of this issue anymore (due to moved issues)

[luxas]

Yes, thanks.
Closing...