self hosted etcd has a circular dependency on service ip.

[coresolve]

With self hosted etcd we see that significant memory pressure on the masters will cause the cluster to become unstable.

This will mean that the kubelet will eventually mark all the etcd pods unhealthy and thus the kube-proxy will rejecting packets headed for the kube-system/etcd-service:client

At this point the apiserver is unable to start as it is configured to connect to etcd cluster with the service ip address of the self hosted cluster. The service continues to have no health endpoints because the kubelet can't talk to the apiserver and tell it that the service has healthy endpoints.

So now we have a healthy etcd cluster and no way to update the record in etcd via the apiserver.

[chancez]

coreos/etcd-operator#622 seems related.

[hongchaodeng]

Tolerate unreadiness is not enough. We also need to take into account restarts.

[hongchaodeng]

From this comment: kubernetes/kubernetes#25283 (comment)

Seems the tolerate-unready feature might be able to tolerate restart (rolling upgrade). We need to test out though

[coresolve]

Further when you restart a node in this condition we aren't checkpointing the sh etcd cluster. So there is no way to get the cluster back in a reboot.

[hongchaodeng]

Filed upstream issue: kubernetes/kubernetes#47880

[hongchaodeng]

We are going to enable alpha annotation TolerateUnreadyEndpointsAnnotation in etcd service. Will submit a PR shortly.