tls failure in cluster set up, v0.5.1, etcd 3.2 #1384
Comments
The extension doesn't affect the file's nature. But it is a key in the secret: |
|
@hongchaodeng so the name also matters? For me the ca was same for peer, server and client, so I used ca.pem for all |
|
@raoofm Just copy it to different names. We did the same in bootkube. Just by design we shouldn't couple them. |
|
@hongchaodeng the etcd pods fail and continously restart.
logs from etcd-dev-cluster-0000 logs from etcd-dev-cluster-0001 logs from etcd-dev-cluster-0036 logs from etcd-dev-cluster-0037
|
|
copying complete logs etcd-dev-cluster-0088 etcd-dev-cluster-0087 |
|
@xiang90 @hongchaodeng any suggestions ? |
Seems like TLS cert problem.
Can you provide reproduce steps? Actually can you try with 3.1 first? We haven't tested 3.2 yet... |
|
@hongchaodeng @xiang90 works with 3.1.8 but not with 3.2.5 I would like to use latest 3.2.6, can your team spend some time to see what is the breaking change in tls and what the end user need to change in cert generation. |
|
@raoofm Have you tried your configuration and see if it works with 3.1 first? |
|
yes that is what I confirmed
|
|
yes I tried and it works with 3.1.8 |
|
Confirmed the same issue with etcd "3.2.5".
This looks suspicious. |
|
@raoofm |
|
ok will do |
hongchaodeng
changed the title
|
@hongchaodeng any update on making etcd-operator compatible with v3.2 etcd |
|
fixed in 0.7.0+ |
Used below to generate self signed certs and create k8s secrets
https://coreos.com/os/docs/latest/generate-self-signed-certificates.html
https://github.com/coreos/etcd-operator/blob/master/doc/user/cluster_tls.md
Logs from etcd-operator
time="2017-09-08T16:18:05Z" level=info msg="etcd-operator Version: 0.5.1"
time="2017-09-08T16:18:05Z" level=info msg="Git SHA: cf7d8d5"
time="2017-09-08T16:18:05Z" level=info msg="Go Version: go1.8.3"
time="2017-09-08T16:18:05Z" level=info msg="Go OS/Arch: linux/amd64"
time="2017-09-08T16:18:23Z" level=info msg="Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"etcd-operator", Name:"etcd-operator", UID:"073b648c-8da3-11e7-8ce6-0637473fabae", APIVersion:"v1", ResourceVersion:"32276337", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' etcd-operator-3603461089-jd754 became leader"
time="2017-09-08T16:18:23Z" level=info msg="finding existing clusters..." pkg=controller
time="2017-09-08T16:18:23Z" level=error msg="cluster failed to setup: tls: failed to find any PEM data in certificate input" cluster-name=etcd-dev-cluster pkg=cluster
time="2017-09-08T16:18:23Z" level=info msg="starts running from watch version: 32276337" pkg=controller
time="2017-09-08T16:18:23Z" level=info msg="start watching at 32276337" pkg=controller
time="2017-09-08T16:18:23Z" level=warning msg="fail to handle event: ignore failed cluster (etcd-dev-cluster). Please delete its CR" pkg=controller
time="2017-09-08T16:20:00Z" level=info msg="apiserver closed watch stream, retrying after 5s..." pkg=controller
time="2017-09-08T16:20:05Z" level=info msg="start watching at 32276338" pkg=controller
ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"server": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
ca-csr.json
{
"CN": "Company CA for etcd-operator",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "US",
"L": "Some City",
"ST": "Some State",
"O": "Some Company",
"OU": "Security",
"OU": "Development"
}
]
}
server.json
{
"CN": "etcdServer",
"hosts": [
"*.etcd-dev-cluster.etcd-operator.svc",
"etcd-dev-cluster-client.etcd-operator.svc",
"localhost"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "US",
"L": "Some City",
"ST": "Some State",
"O": "Some Company",
"OU": "Security",
"OU": "Development"
}
]
}
peer.json
{
"CN": "etcdPeer",
"hosts": [
"*.etcd-dev-cluster.etcd-operator.svc"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "US",
"L": "Some City",
"ST": "Some State",
"O": "Some Company",
"OU": "Security",
"OU": "Development"
}
]
}
client.json
{
"CN": "client",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "US",
"L": "Some City",
"ST": "Some State",
"O": "Some Company",
"OU": "Security",
"OU": "Development"
}
]
}
Commands used
kubectl create secret generic etcd-peer-tls --from-file=ca.pem --from-file=peer.pem --from-file=peer-key.pem
kubectl create secret generic etcd-server-tls --from-file=ca.pem --from-file=server.pem --from-file=server-key.pem
kubectl create secret generic etcd-client-tls --from-file=ca.pem --from-file=client.pem --from-file=client-key.pem
The only difference that I could see is cfssl spits out pem encoded and the example in etcd-operator cluster_tls setup have .crt and .key file extension (it shouldn't matter as the doc says they should be pem-encoded)
The text was updated successfully, but these errors were encountered: