TestCtlV3DelTimeout: ctlV3Put errors with 'connection is closing'

[gyuho]

https://semaphoreci.com/coreos/etcd/branches/pull-request-8580/builds/1

--- FAIL: TestCtlV3DelTimeout (0.62s)
	ctl_v3_kv_test.go:279: delTest #0-2: ctlV3Put error (read /dev/ptmx: input/output error (expected "OK", got ["Error: rpc error: code = Unavailable desc = grpc: the connection is closing\r\n"]))

Breaks since #8545.

Update:

---
../bin/etcdctl3 --endpoints=http://localhost:20000 --dial-timeout=0s put sample value

newSimpleBalancer close(sb.downc)
################ updateNotifyLoop downc == nil ()
=======notifyAddrs b.notifyCh <- [{localhost:20000 <nil>}]

newRetryWrapper rpc error: code = Unavailable desc = there is no address available
 
newRetryWrapper c.balancer.next() 1

---------- next b.updateAddrsC <-notifyNext 1
---------- next b.updateAddrsC <-notifyNext 2
---------- next <-downc
 
newRetryWrapper c.balancer.next() 2

################ updateNotifyLoop downc == nil <-b.updateAddrsC () (msg 1)
=======notifyAddrs notifyNext []grpc.Address{}
=======notifyAddrs b.notifyCh <- [{localhost:20000 <nil>}]
################ updateNotifyLoop downc == nil ()
=======notifyAddrs b.notifyCh <- [{localhost:20000 <nil>}]
up {localhost:20000 <nil>}
up close(b.upc)
up pinAddr {localhost:20000 <nil>}

newRetryWrapper <-notify

################ updateNotifyLoop downc == nil <-upc ()
################ updateNotifyLoop upc == nil (localhost:20000)
################ updateNotifyLoop upc == nil b.notifyCh <- localhost:20000
OK

---
../bin/etcdctl3 --endpoints=http://localhost:20000 --dial-timeout=0s put key1 val1

newSimpleBalancer close(sb.downc)
################ updateNotifyLoop downc == nil ()
=======notifyAddrs b.notifyCh <- [{localhost:20000 <nil>}]

newRetryWrapper rpc error: code = Unavailable desc = there is no address available

newRetryWrapper c.balancer.next() 1

---------- next b.updateAddrsC <-notifyNext 1
---------- next b.updateAddrsC <-notifyNext 2
up {localhost:20000 <nil>}
up close(b.upc)
up pinAddr {localhost:20000 <nil>}
################ updateNotifyLoop downc == nil <-b.updateAddrsC () (msg 1)
---------- next <-downc

newRetryWrapper c.balancer.next() 2

newRetryWrapper <-notify

up close(b.downc)
=======notifyAddrs notifyNext []grpc.Address{}
up error {localhost:20000 <nil>} (grpc: the connection is drained)

newRetryWrapper rpc error: code = Unavailable desc = grpc: the connection is closing

=======notifyAddrs b.notifyCh <- [{localhost:20000 <nil>}]

newRetryWrapper c.balancer.next() 1

---------- next b.updateAddrsC <-notifyNext 1
################ updateNotifyLoop downc == nil ()
=======notifyAddrs b.notifyCh <- [{localhost:20000 <nil>}]
################ updateNotifyLoop downc == nil <-b.updateAddrsC () (msg 1)
=======notifyAddrs notifyNext []grpc.Address{}
---------- next b.updateAddrsC <-notifyNext 2
---------- next <-downc

newRetryWrapper c.balancer.next() 2

Error: rpc error: code = Unavailable desc = grpc: the connection is closing


--- FAIL: TestCtlV3DelTimeout (0.36s)

b.notifyAddrs(notifyReset) needs fix, it seems.