randomize etcd member name

[hongchaodeng]

Currently we have a counter in member name to increment when adding new members. This could lose track though, e.g. operator restarts and highest member crashes, all pods are deleted and restore operator knows nothing.

When two etcd pods have the same name, it could lead to bad results -- same DNS record returns two different IPs. In k8s, if we delete a pod, it is asynchronous. And we can't guarantee that when we recreate another new pod with the same name, there would only be exactly one pod of the same. Taking these facts, we can see the problems here. In fact, we have seen real issues like #1825.

We should randomize etcd member name just like ReplicaSet does to each replica pod. This way we can prevent two etcd members from having the same name.

[alexandrem]

How will we build the list of peers (no discovery) and have TLS work with ALT names using random names?

[hongchaodeng]

@alexandrem
Member DNS names have the same subdomain:

etcd-operator/example/tls/certs/peer.json

Line 4 in ffbd359

"*.example.default.svc",

[alexandrem]

Right, so it's good with wildcards.

But what about initial peer list?

[hongchaodeng]

But what about initial peer list?

Sorry. I don't understand what's this concern about? Mind give an example?

[alexandrem]

Sorry I might have in mind the case where we change the restartPolicy of pod members, so possibly this doesn't apply at the very moment.

Currently, I assume that if a member is unhealthy then the operator will replace it with a fresh pod and therefore can pass the recent active pod members as peers list.

On the other hand, if we have a restartPolicy: Always then I assume that the pod member who gets rescheduled or restarted for some reasons will have its old peers list configuration and won't be able to join back the cluster.

That is, if members changed since this pod creation.

Is this correct?

[hongchaodeng]

etcd-operator will have the global membership knowledge and configure the peer list for each etcd pod.

But from my understand, even if membership changes during pod replacement, etcd member will still have the logs (data) and sync with leader to catch up missing knowledge.

[alexandrem]

Ok, I don't have that much operational knowledge of etcd.

I was under the impression that when using static configuration the --initial-cluster parameter values of a member had to match exactly the active members of the cluster, including this new peer name, otherwise it wouldn't sync.