etcd3 fails to shut down on sigint/sigterm when grpc-gateway has active watches

[jamesyonan]

Running a single-node cluster from the shell using 3.2.0-rc.1+git. I have a local client connect to the JSON grpc-gateway using POST /v3alpha/watch in "streaming" mode, where the HTTP session is long-term and each new watch event comes over the wire as a new JSON-encoded HTTP chunk. This part works fine. However if I try to ctrl-c the etcd process while the watch is still active, the process hangs in a loop with the below errors. The only way to get it to terminate is either sigkill or terminating the watch client.

^C2017-07-06 17:02:31.710640 N | pkg/osutil: received interrupt signal, shutting down... 2017-07-06 17:02:31.710981 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: getsockopt: connection refused"; Reconnecting to {127.0.0.1:2379 <nil>} 2017-07-06 17:02:32.711047 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: getsockopt: connection refused"; Reconnecting to {127.0.0.1:2379 <nil>}

[gyuho]

Yes, this is expected since we use https://godoc.org/google.golang.org/grpc#Server.GracefulStop. The server will block on GracefulStop with active watchers https://github.com/coreos/etcd/blob/master/embed/etcd.go#L196.

Same happens with clientv3

e.g.

wch := cli.Watch(context.Background(), "foo")
<-wch

/cc @heyitsanthony @xiang90

[heyitsanthony]

@gyuho Is there some way the server can stop the client listener and cut the connections instead of waiting for all clients to disconnect on their own? If clients can keep the server from shutting down indefinitely and there's no way to force the clients to change endpoints, then upgrades would have to kill -9 the etcd process (or jump through other hoops to move the clients) to make forward progress, which seems bad.

[gyuho]

@heyitsanthony You are right. I will look into that.

[jamesyonan]

I agree that it's generally a bad idea for servers to fail to respond to a shutdown command just because clients are connected. There's a security component to this as well, as @heyitsanthony mentioned with updates. If a client can prevent a server from shutting down, just by staying connected to it, it could effectively stop a security update.

James