Readiness probe optionally not affecting receiving traffic via service

[hongchaodeng]

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Readiness probe. Service. Stateful pod.

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

FEATURE REQUEST

When etcd starts, it has a bootstrap phase that talks to other peers:

Start -> bootstrapping -> running/serving

If "bootstrap" phase failed, it is the same as dead because no data is available, no quorum information is known.

However, if we start etcd in a Kubernetes Pod, even if it starts running, it doesn't mean etcd is running. it is important for the operator to know the status of the etcd member in this phase. In other words, if we starts an etcd pod, how could we tell etcd is running?

The closest way to achieve that seems to be ReadinessProbe. We can have it probing etcd to know whether it's "ready".

However, using ReadinessProbe has a problem. We reserve a service name for each etcd pod in order to form a known cluster, e.g. "m1=m1-service:2380,m2=m2-service:2380,m3=m3-service:2380", at starting etcd. If a pod isn't ready, then it couldn't receive traffic, and thus members couldn't contact each other.

A potential solution is to make Readiness decoupled from "receiving traffic". Ready is a special state between "Pod starts running" and "Pod starts serving".

[hongchaodeng]

ref: #39363

[smarterclayton]

Usually this is via two services, one with "tolerate unready endpoints" true (for member detection) and one for serving traffic.

[hongchaodeng]

Closing this to merge discussion into #39363