request help: How to configure TLS connections for etcd

[tb51cx]

Issue description

helm安装apisix-ingress时，如果用外部ETCD。并且外部ETCD是启用SSL的。那么除了--set etcd.enabled=false和 --set etcd.host以外应该还要指定ca cert 和key请问能给个使用外部ETCD并且ETCD开始SSL的例子吗？

Environment

• your apisix-ingress-controller version (output of apisix-ingress-controller version --long):
• your Kubernetes cluster version (output of kubectl version):
• if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a):

[tao12345666333]

In fact, this question has nothing to do with Ingress. So i moved it and changed the title.

Currently we are using the helm chart of bitnami/etcd, please check its documentation.
https://github.com/bitnami/charts/tree/master/bitnami/etcd#parameters

And https://etcd.io/docs/v3.2/op-guide/security/

[tb51cx]

ETCD为已经在本机安装好的集群，配置如下:
systemctl cat etcd

/etc/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd
ExecStart=/opt/kube/bin/etcd
--name=etcd-172.18.188.208
--cert-file=/etc/kubernetes/ssl/etcd.pem
--key-file=/etc/kubernetes/ssl/etcd-key.pem
--peer-cert-file=/etc/kubernetes/ssl/etcd.pem
--peer-key-file=/etc/kubernetes/ssl/etcd-key.pem
--trusted-ca-file=/etc/kubernetes/ssl/ca.pem
--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem
--initial-advertise-peer-urls=https://172.18.188.208:2380
--listen-peer-urls=https://172.18.188.208:2380
--listen-client-urls=https://172.18.188.208:2379,http://127.0.0.1:2379
--advertise-client-urls=https://172.18.188.208:2379
--initial-cluster-token=etcd-cluster-0
--initial-cluster=etcd-172.18.188.208=https://172.18.188.208:2380,etcd-172.18.188.205=https://172.18.188.205:2380,etcd-172.18.188.206=https://172.18.188.206:2380
--initial-cluster-state=new
--data-dir=/var/lib/etcd
--wal-dir=
--snapshot-count=50000
--auto-compaction-retention=1
--auto-compaction-mode=periodic
--max-request-bytes=10485760
--quota-backend-bytes=8589934592
Restart=always
RestartSec=15
LimitNOFILE=65536
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

证书内容如下：
ll /etc/kubernetes/ssl
total 40
-rw-r--r-- 1 root root 1679 Mar 2 17:05 aggregator-proxy-key.pem
-rw-r--r-- 1 root root 1383 Mar 2 17:05 aggregator-proxy.pem
-rw-r--r-- 1 root root 1675 Mar 2 17:05 ca-key.pem
-rw-r--r-- 1 root root 1302 Mar 2 17:04 ca.pem
-rw-r--r-- 1 root root 1675 Mar 2 17:04 etcd-key.pem
-rw-r--r-- 1 root root 1428 Mar 2 17:04 etcd.pem
-rw-r--r-- 1 root root 1679 Mar 2 17:06 kubelet-key.pem
-rw-r--r-- 1 root root 1452 Mar 2 17:06 kubelet.pem
-rw-r--r-- 1 root root 1679 Mar 2 17:05 kubernetes-key.pem
-rw-r--r-- 1 root root 1736 Mar 2 17:05 kubernetes.pem

测试证书+ETCD状态如下：
ETCDCTL_API=3 etcdctl
--endpoints=https://172.18.188.208:2379
--cacert=/etc/kubernetes/ssl/ca.pem
--cert=/etc/kubernetes/ssl/etcd.pem
--key=/etc/kubernetes/ssl/etcd-key.pem
endpoint health
https://172.18.188.208:2379 is healthy: successfully committed proposal: took = 11.030749ms

创建K8S证书
cp /etc/kubernetes/ssl/etcd.pem /root/zhengshu/tls.crt
cp /etc/kubernetes/ssl/etcd-key.pem /root/zhengshu/tls.key

kubectl create secret tls etcd -n ingress-apisix --cert=/root/zhengshu/tls.crt --key=/root/zhengshu/tls.key --dry-run=client -o yaml > etcd-ssl.yaml
kubectl apply -f etcd-ssl.yaml
kubectl describe secret etcd -n ingress-apisix
Name: etcd
Namespace: ingress-apisix
Labels:
Annotations:

Type: kubernetes.io/tls

Data

tls.crt: 1428 bytes
tls.key: 1675 bytes

apisix helm安装命令如下:
helm install apisix apisix/apisix
--set gateway.type=NodePort
--set ingress-controller.enabled=true
--namespace ingress-apisix
--set ingress-controller.config.apisix.serviceNamespace=ingress-apisix
--set etcd.enabled=false
--set etcd.auth.tls.enabled=true
--set etcd.host={https://172.18.188.208:2379\,https://172.18.188.205:2379\,https://172.18.188.206:2379}
--set etcd.auth.tls.existingSecret=etcd
--set etcd.auth.tls.certFilename=tls.crt
--set etcd.auth.tls.certKeyFilename=tls.key

POD报错内容如下:
kubectl logs apisix-7c6d459dbd-vlxbb -n ingress-apisix
/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init

WARNING: using fixed Admin API token has security risk.
Please modify "admin_key" in conf/config.yaml .

/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
Warning! Request etcd endpoint 'https://172.18.188.208:2379/version' error, certificate verify failed, retry time=1
request etcd endpoint 'https://172.18.188.208:2379/version' error, certificate verify failed
request etcd endpoint 'https://172.18.188.205:2379/version' error, certificate verify failed
request etcd endpoint 'https://172.18.188.206:2379/version' error, certificate verify failed
all etcd nodes are unavailable
Warning! Request etcd endpoint 'https://172.18.188.208:2379/version' error, certificate verify failed, retry time=2
Warning! Request etcd endpoint 'https://172.18.188.205:2379/version' error, certificate verify failed, retry time=1
Warning! Request etcd endpoint 'https://172.18.188.205:2379/version' error, certificate verify failed, retry time=2
Warning! Request etcd endpoint 'https://172.18.188.206:2379/version' error, certificate verify failed, retry time=1
Warning! Request etcd endpoint 'https://172.18.188.206:2379/version' error, certificate verify failed, retry time=2

这样看来还是证书的问题，请问我set参数还是有问题吗？

[tao12345666333]

please see apache/apisix#3371
apache/apisix#3270
apache/apisix#3225