Network-partition aware health service

[gyuho]

clientv3 health balancer should be able to reason about network-partitions with keepalive HTTP/2 ping.

#8660 makes balancer aware of network partitions on time-out errors.
But only handles the case when client specifies context time-out.

We can do better.

1. Client sends linearized requests with context time-out x
   • or no time-out with context.Background
2. Client configures keepalive HTTP/2 ping time-out y, where y < x
3. Balancer pins endpoint A in 3-node cluster
4. Member A becomes isolated
5. Linearized request to A blocks until time-out x
   • blocks forever if requested with context.Background

When y < x, keepalive pings should detect that member A cannot reach other members.
Then trigger endpoint switch before time-out x elapse.

[gyuho]

/cc @jpbetz

[jpbetz]

/cc @wojtek-t

[gyuho]

We still plan to do this (maybe in v3.5).

Merging with #8022, since both server and client need reason about network partitions. Current HTTP/2 ping client and server do not reason about network partition.

There are cases where the client needs to know that the server is responding to requests. If the server is not responding (but possibly still accepting connections), the balancer should try another endpoint. For instance, a watch may be disconnected but the connection will stay open and hang instead of connecting to another member.

This heartbeat mechanism is somewhat weaker in that it doesn't need a leader, but there still needs to be some kind of polling to avoid waiting indefinitely on a disconnected socket. The client implementations for heartbeat/lost leader reconnect can both poll or lost leader can depend on heartbeat to cover the wait case.

ref. #7321

[gyuho]

Another use case kubernetes/kubernetes#59848 (comment)

This can happen on a singleton master using an HA etcd cluster:

apiserver is talking to member 0 in the cluster
apiserver crashes and watch cache performs its initial list at RV=10
kubelet deletes pod at RV=11
a new pod is created at RV=12 on a different node
kubelet crashes
apiserver becomes partitioned from the etcd majority but is able to reestablish a watch from the partitioned member at RV=10
kubelet contacts the apiserver and sees RV=10, launches the pod

[gyuho]

And just to clarify, currently require leader metadata must be passed to prevent this:

wch :=  cli.Watch(clientv3.WithRequireLeader(context.Background()), "foo")
// will be closed when the node loses leader

wch :=  cli.Watch(context.Background(), "foo")
// block forever, even if a node loses leader

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.