Using your own certificates when building a new cluster

[afriedma]

I can't find any documentation where should I put my ca.pem and ca.key to get the kubespray to use them to generate /etc/kubernetes/pki and /etc/kubernetes/ssl and etcd certificates,

I see that I can set cert_management to none, but I don't see how my certs would be used when I run cluster.yml. My target configuration would be 2 master 3 worker and 3 etcd, for the total of 3 node cluster.

[mirwan]

Good point, there is no doc about that, I think I could work on something...
In the meantime:
etcd:

• when etcd_kubeadm_enabled=false (default), cert and key should be put in /etc/ssl/etcd/ssl/ca.pem and /etc/ssl/etcd/ssl/ca-key.pem
• when etcd_kubeadm_enabled=true, cert and key should be put in /etc/kubernetes/ssl/etcd/ca.crt and /etc/kubernetes/ssl/etcd/ca.key

k8s:

• cert and key should be put in /etc/kubernetes/ssl/ca.crt and /etc/kubernetes/ssl/ca.key

[afriedma]

Thank you for reply, just to clarify, my certs would be copied by me manually to those locations after cluster.yml is done?

If that's the case, do I need to generate node-node1.pem, node-node1-key.pem manually as well, just as an example?

I thought there would be a directory in the kubespray for me to copy those cirtificates to, and all the generation would be automated, may be I am asking for too much:) just trying to understand the workflow.

[mirwan]

The cert and key of the CAs must be copied to the servers prior to running the cluster.yml playbook.
Apart from that, everything is automated:
If kubespray (...) doesn't find any CA cert and key, it will generate self-signed CA cert and key.
Then it will use the CA cert and key (previously generated or simply copied) to generate the server/client certs for the components.

[mirwan]

BTW I forgot to mention the front-proxy CA (cert and key are located in /etc/kubernetes/ssl/front-proxy-ca.crt and /etc/kubernetes/ssl/front-proxy-ca.key) which is different from the /etc/kubernetes/ssl/ca.crt and /etc/kubernetes/ssl/ca.key.
It's the same workflow, both cert and key for front-proxy CA must be copied prior to runnning cluster.yml, otherwise they will be generated as self-signed

[afriedma]

Thank you for detailed explanations. I will revert back, if I run into any issues.

[shukla2009]

It will be very useful if we can specify certs while running cluster.yml

[pratap-redi]

@mirwan I tried the same, copied my certs and keys as /etc/ssl/etcd/ssl/ca.pem, /etc/ssl/etcd/ssl/ca-key.pem (as I see etcd_kubeadm_enabled=false), /etc/kubernetes/ssl/ca.crt, /etc/kubernetes/ssl/ca.key , but cluster build fails with the below error, Any thoughts?

FAILED! => {"changed": false, "cmd": "/usr/local/bin/etcdctl --endpoints=https://172.27.65.117:2379,https://172.27.65.116:2379,https://172.27.65.131:2379 cluster-health | grep -q 'cluster is healthy'", "delta": "0:00:00.040758", "end": "2020-03-30 02:32:58.636732", "msg": "non-zero return code", "rc": 1, "start": "2020-03-30 02:32:58.595974", "stderr": "Error: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 172.27.65.117:2379: connect: connection refused\n; error #1: dial tcp 172.27.65.131:2379: connect: connection refused\n; error

[mirwan]

@pratap-redi Can you check the output of openssl x509 cmd on the generated certs ? Are they issued by your CA as expected?
Connection refused is not related to certs (unless certs are not wrong/missing)

[pratap-redi]

HI @mirwan Thanks for the response.

I am building a 4 node cluster and In the certs I see all the 4 nodes mentioned as X509v3 Subject Alternative Name.

Also I tried to run the below to check the status, it returns the same.

etcdctl --endpoints https://172.27.65.117:2379 --ca-file=/etc/ssl/etcd/ssl/ca.pem --cert-file=/etc/ssl/etcd/ssl/ca.pem --key-file=/etc/ssl/etcd/ssl/ca-key.pem --debug cluster-health
Cluster-Endpoints: https://172.27.65.117:2379
cURL Command: curl -X GET https://172.27.65.117:2379/v2/members
cluster may be unhealthy: failed to list members
Error: client: etcd cluster is unavailable or misconfigured; error #0: client: endpoint https://172.27.65.117:2379 exceeded header timeout

error #0: client: endpoint https://172.27.65.117:2379 exceeded header timeout

[mirwan]

cert-file=/etc/ssl/etcd/ssl/ca.pem key-file=/etc/ssl/etcd/ssl/ca-key.pem? should be server cert/key not ca/ca key here I think

[pratap-redi]

HI @mirwan It seems our company doesn't allow IP addresses in the subject alternative names of the certificates they areonly including FQDN of the server.

I tried to install the cluster manually with kubeadm and I did kubeadm init --config=config.yaml and in the config.yaml I specified the :2379 instead of IP adress, It worked.

So Where I can edit in the kubespray to use FQDN instead of IP ? in the etcd/apiserver endpoints

[mirwan]

IP SANs are not subject to conditions in roles/etcd/templates/openssl.conf.j2 or roles/kubernetes/master/tasks/kubeadm-setup.yml, so far there is no way not to add IP as SANs. You could provide a PR to add this option but I don't think not having IP in SANs would work in some cases (apiserver certs) ...

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[khmarochos]

Excuse me, I'd like to add a question.

What if my CA key is DES-encrypted and needs a passphrase? How to provide with the passphrase to kubespray?

[k8s-ci-robot]

@Melnik13: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.