About client closing gRPC stream on receiving HTTP/2 GOAWAY

[krapie]

Hi, I'm currently searching for how to close connection when client receives HTTP/2 GOAWAY, and I'm confused with how this actually works. So I'm leaving this question in gRPC-go. My questions may be opaque, so my apologies for my lack of knowledge.

Context

I'm using gRPC server-side streaming, and I'm also using envoy for hash-based load balancing. But I've faced split-brain of gRPC stream(long-lived connection) when upstream host set changes. For more information, follow: envoyproxy/envoy#26459.

I'm aware that I can set gRPC's MAX_CONNECTION_AGE to resolve this issue, Ref: Using gRPC for Long-lived and Streaming RPCs.

Question

But how do I manage client to close connection on GOAWAY?

I've searched for HTTP/2 GOAWAY RFC Spec and gRPC-go's http2_server with http2_client codes. And these are what I understood:

• RFC states that server sends GOAWAY to initiate shutdown of a connection, and client can decide to retry connection on receiving GOAWAY.
• http2_server in gRPC's transport layer sends two GOAWAY frames. with first GOAWAY with stream ID of MaxUInt32 and second GOAWAY with last processed stream id.

  grpc-go/internal/transport/http2_server.go

  Line 1364 in df82147

  // For a graceful close, send out a GoAway with stream ID of MaxUInt32,

• Then http2_client receives these two frames and performs stream close when stream id is greater than the last_stream_id and smaller than the previous last_stream_id.

  grpc-go/internal/transport/http2_client.go

  Line 1306 in df82147

  // All streams with IDs greater than the GoAwayId

I understood as gRPC is handling GOAWAY as HTTP/2 layer under in gRPC stacks, in http2_server and http2_client, but I'm confused about:

• Does http2_server and http2_client completely handles GOAWAY? Then I don't have to handle GOAWAY and close connection manually?
• If that is http2_server and http2_client completely handles GOAWAY, why this statement below does not become valid when GOAWAY is sent to close stream? e.g: If the stream id that I want to close is 5, then should id(last_stream_id) be smaller than 5? (which does not make sense because last_stream_id will be 5, as last processed stream)

  grpc-go/internal/transport/http2_client.go

  Line 1322 in df82147

  if streamID > id && streamID <= upperLimit {

Since http2_client is not closing stream connection even receiving GOAWAY, I tried to capture HTTP/2 frames and close connection manually. But I don't see any APIs to get HTTP/2 frames.

To summarize my questions:

1. What is the normal behavior of closing stream connection on GOAWAY in http2_client? (how last_stream_id should be set in order to close the stream?)
2. Is there any way to manually capture GOAWAY and reset(close) connection using gRPC go-sdk?

If you are wondering why I'm trying to close stream manually, follow: yorkie-team/yorkie#526 for more contexts.

[easwars]

Does http2_server and http2_client completely handles GOAWAY? Then I don't have to handle GOAWAY and close connection manually?

Yes gRPC takes care of handling GOAWAY frames. You should not have to do anything from your client application to close the connection upon receipt of a GOAWAY frame.

If that is http2_server and http2_client completely handles GOAWAY, why this statement below does not become valid when GOAWAY is sent to close stream? e.g: If the stream id that I want to close is 5, then should id(last_stream_id) be smaller than 5? (which does not make sense because last_stream_id will be 5, as last processed stream)

There is an inherent race condition between the server sending GOAWAYs and the client creating new streams. Let's consider the following sequence of events:

• at T0: gRPC server sends first GOAWAY with stream ID set to max uint

• at T1: gRPC client receives this GOAWAY

  • Between T0 and T1, the client has created other streams (S2, S4, S6, S8)
  • After receipt of this first GOAWAY, it stops creating new streams on this connection

• at T2: gRPC server sends the second GOAWAY with stream ID set to S6

  • because S6 is the last stream ID it has seen so far, it has not yet seen S8
  • server will not accept any more streams on this connection

• at T3: gRPC client receives the second GOAWAY, with stream ID set to S6

  • so, the client has to close all streams which have stream ID greater than S6

• gRPC client will wait for existing streams (whose ID is <= S6) to gracefully close, before closing the connection to the server

  What is the normal behavior of closing stream connection on GOAWAY in http2_client? (how last_stream_id should be set in order to close the stream?)

Hope the above explanation helps.

Is there any way to manually capture GOAWAY and reset(close) connection using gRPC go-sdk?

No.

[krapie]

Thank you for your detailed explanation!

Now I understand how GOAWAY works. So GOAWAYis basically telling client to not create further streams and close streams that client created after server sent GOAWAY (now I understand why client is closing stream with bigger stream id than last_stream_id), and client will wait for previous streams to gracefully close before closing connection to the server.

Then, how to gracefully close existings streams after GOAWAY sequence?

gRPC client will wait for existing streams (whose ID is <= S6) to gracefully close, before closing the connection to the server

I've noticed that when closing streams in server-side streaming, the server should knew about when to stop on initial message, so server should close the stream. Ref: grpc/grpc#8023

In general, the assumption of a server-only streaming call is that the initial message from the client contains enough information for the server to know when to stop.

But my server-side streaming rpc is intended to never finish (it is responsible for publishing and subscribing events when client is connected to the server, like chat server).

The only situation when rpc is(should be) closed is when upstream host set changes and split-brain issue happens, so that server have to close dead rpc.

I'm using gRPC server-side streaming, and I'm also using envoy for hash-based load balancing. But I've faced split-brain of gRPC stream(long-lived connection) when upstream host set changes. For more information, follow: envoyproxy/envoy#26459.

To conclude, I just want to gracefully close server-side streaming connection when needed (eg: max_connection_age, idle timeout, close_on_host_set_change, anything that sends GOAWAY).

[easwars]

Then, how to gracefully close existings streams after GOAWAY sequence?

Streams are closed when the server responds to a unary RPC and that response it received by the client. Or in the case of streaming RPCs, the stream on the server is closed when the server application returns from the RPC handler, and this is propagated to the client. The client can also initiate a close from its side by calling CloseSend as defined here: https://pkg.go.dev/google.golang.org/grpc#ClientStream

To conclude, I just want to gracefully close server-side streaming connection when needed (eg: max_connection_age, idle timeout, close_on_host_set_change, anything that sends GOAWAY).

If you want to close the connection from the server-side, you can set the MaxConnectionAge field here: https://pkg.go.dev/google.golang.org/grpc/keepalive#ServerParameters

[krapie]

If you want to close the connection from the server-side, you can set the MaxConnectionAge field here: https://pkg.go.dev/google.golang.org/grpc/keepalive#ServerParameters

I'm already using MaxConnectionAge to gracefully close connection form the server-side, but MaxConnectionAge alone actually did not close connection.

grpc-go/internal/transport/http2_server.go

Line 1173 in 47b3c55

case <-ageTimer.C:

Seems like in http2_server, MaxConnectionAgeGrace is also needed to actually close connection with t.controlBuf.put(closeConnection{}).

MaxConnectionAge is just sending GOAWAY to the client with t.Drain("max_age"), and not performing connection closure.

So I also configured MaxConnectionAgeGrace to forcefully close connection after MaxConnectionAge period for now, but I don't like this is a right way to "gracefully" close connection with no error emitted.

[easwars]

But that is working as intended, and the function and default values of each of those fields are clearly documented.

Do you still think something is not working as documented?

[krapie]

Do you still think something is not working as documented?

I think I misunderstood MaxConnectionAge as to close connection after sending GOAWAY, actually is just sending GOAWAY to stop accepting new streams for graceful close.

Then, since MaxConnectionAge alone is not closing connection, I need to close connection by the method you have mentioned above.

in the case of streaming RPCs, the stream on the server is closed when the server application returns from the RPC handler

So maybe I should return server-side streaming service function periodically?

The client can also initiate a close from its side by calling CloseSend as defined here:

For this method, is there any way to client get notified (I want to set client get notified on GOAWAY but you said there is no way to capture GOAWAY) and just call CloseSend() to close connection?

So my sole question is: is there any way to close connection associated with GOAWAY? (client or server close connection completely after creating/receiving GOAWAY frame?)

[easwars]

Then, since MaxConnectionAge alone is not closing connection, I need to close connection by the method you have mentioned above.

MaxConnectionAge and MaxConnectionAgeGrace, when used together should result in the closing of connections. I don't understand what you mean by "I need to close connection by the method you have mentioned above."

[krapie]

I'm actually using MaxConnectionAge and MaxConnectionAgeGrace to close connection for temporary solution for my issue.

But I thought MaxConnectionAgeGrace was not a normal way to close connection, since this option forcibly closes connection. And I thought there should be other ways to normally(gracefully) close connection.

I don't understand what you mean by "I need to close connection by the method you have mentioned above.

I meant that I might have to use the methods you have mentioned earlier.

Streams are closed when the server responds to a unary RPC and that response it received by the client. Or in the case of streaming RPCs, the stream on the server is closed when the server application returns from the RPC handler, and this is propagated to the client. The client can also initiate a close from its side by calling CloseSend as defined here: https://pkg.go.dev/google.golang.org/grpc#ClientStream

If you want to close the connection from the server-side, you can set the MaxConnectionAge field here: https://pkg.go.dev/google.golang.org/grpc/keepalive#ServerParameters

[easwars]

And I thought there should be other ways to normally(gracefully) close connection

Graceful close of connections wait for existing streams to be closed before the connection is closed. If your server RPC handler never returns, then existing streams will not be closed, and therefore graceful connection close will not happen.

But I thought MaxConnectionAgeGrace was not a normal way to close connection.

Please see https://github.com/grpc/proposal/blob/master/A9-server-side-conn-mgt.md for more details.

[krapie]

Now I understand the full context of this issue.

Graceful close of connections wait for existing streams to be closed before the connection is closed. If your server RPC handler never returns, then existing streams will not be closed, and therefore graceful connection close will not happen.

Since my server-side streaming rpc never returns, there will be no "graceful close" of connection, so even when GOAWAY is sent, there will be no additional graceful connection close.

I might need to add timer in my rpc to perform graceful connection close when timer expires, combined with MaxConnectionAge (eg: MaxConnectionAge set to 60, rpc timer to 70, and MaxConnectionAgeGrace to 80 or so).

[easwars]

@krapie : It does look like this issue can be closed now. Please let us know if you have other open questions. Thanks.