How to graceful shutdown client or server when connection still have active streams

[firebook]

I create a bidirectional streaming RPC.
In server side, I call onNext of responseStreamObserver continuous, then I try to invoke server.shutdown.
But server will never shutdown for there is active stream.
Only the active stream send end frame will trigger stream close and then trigger the connection close and server shutdown.
Also, when I try to shutdown subChannel in client side, connection will never close for there is active stream.

Both side will send a GO_AWAY frame, but in gRPC layer, requestStream can not receive this status, so stream can not stop by send onComplete.
Why not notify requestStream when receive a GO_AWAY frame? And then stream will know it's time to do onComplete.

[ejona86]

The normal server shutdown process would be:

server.shutdown();
server.awaitTermination(1, TimeUnit.MINUTES); // Whatever value you want
server.shutdownNow();

That pattern has a speedy shutdown if all RPCs complete quickly but also an upper bound of 1 minute (as written). shutdownNow() cancels any RPCs that didn't complete, so it is just "reasonably" graceful. The same pattern can be used on client-side, although it is more common to know that the channel is not being used and just call channel.shutdownNow() initially.

I generally recommend purposefully closing long-lived streams after they reach a certain age to allow the client to re-balance to another server. And long-lived streams typically have to suffer disruptions do to changing networks. So a cancel frequently is not hard to handle.

A GOAWAY on a connection doesn't mean that connection has to die right now. The connection can live for a day after the GOAWAY. There's multiple reasons to use it and not all of them (e.g., MAX_CONNECTION_AGE) would purposefully kill the RPC. The degenerate case for something like MAX_CONNECTION_AGE is "1 TCP connection per long-lived RPC," which isn't that high of a cost for those with HTTP/1 background.

I have considered having a notification on GOAWAY, but it isn't quite that simple. Initially I thought that the notification should be server-side only, as you really don't want to require all your clients to have proper clean-up logic. But that breaks when using a reverse proxy in front of the server. With a proxy, the connection from the proxy to the backend may have an unbounded lifetime and so only rarely comes into play. Instead, the connection between the client and the proxy matters more, so then we need the notification on client-side. But that breaks if there are two proxies (client → proxy 1 → proxy 2 → backend) between the client and server and the connection between the proxies sees GOAWAY.

grpc/grpc#26703 is a related discussion.

[ejona86]

Seems like this is resolved. If not, comment, and it can be reopened.