etcd2 client doesn't cycle to next cluster member in some error conditions

[rrati]

In Openshift we have seen errors like:

apiserver received an error that is not an metav1.Status: dial tcp :2379: getsockopt: connection refused

This can happen when the etcd cluster leader goes away. In my testing I got it to occur when I manually shut down the etcd process on the node that is acting as the cluster leader. When this condition occurs, the etcd client continually tries to hit the down etcd leader and doesn't cycles to the other available etcd servers for a very long time (10+ minutes).

Looking at the code in client/client.go it seems there are some error cases where the client won't attempt to contact the next server. The OneShot case definitely won't cycle in error conditions.

Is this expected? It seems like you would want the client to cycle to the next cluster member for each request regardless of whether the previous one was a failure or a success.