Server might not send GOAWAY when some streams are being processed

[pderop]

Motivation:

Sometimes, when initiating a graceful shutdown on an HTTP2 server, it may happen that some connections are closed without sending GOAWAY.

This may (not always) happen under the following conditions:

• the server is configured with H2/HTTP11 or H2C/HTTP11
• some streams are currently being processed on the server
• at this point a graceful shutdown is initiated

Under these conditions, the channelGroup that is associated to the HttpServer contains both stream channel as well as their corresponding parent socket channels. So, when the graceful shutdown is initiated, the ServerTransport.disposeNow(Duration) method is iterating on all channels registered in the channel group.

Let's say we have one http connection, and one currently active stream with id=3, so in the channelGroup, we have something like:

0 = NioSocketChannel [id: 0xfce41a80, L:/xxx.xxx.xxx.xxx:58780 - R:/xxx.xxx.xxx.xxx:58782]
1 = Http2MultiplexHandler$Http2MultiplexHandlerStreamChannel [id: 0xfce41a80, L:/xxx.xxx.xxx.xxx:58780 - R:/xxx.xxx.xxx.xxx:58782](H2 - 3)

Now, assume that during the first iteration over the channelGroup, disposeNow first finds the first channel (the parent channel: NioSocketChannel with id: 0xfce41a80 ), then this parent channel will be registered in the channelsToMono map (with an empty ArrayList).

Now, for the second iteration, disposeNow method is now getting the second channel (the stream channel: Http2MultiplexHandler$Http2MultiplexHandlerStreamChannel with id: 0xfce41a80. In this case the parent won't be inserted in the channelsToMono, because the parent has already been inserted during the first iteration.

So, during the second iteration, we won't close the stream here.

But the stream is being processed and there is a channel operation which is active so the current operation will be added to the list here, and because of this the parent channel also won't be closed here, and the socket will be closed when the JVM exits without having sent a GOAWAY message.

This PR first modifies the existing doTestGracefulShutdown in HttpServerTests, and the method is now checking that GOAWAY are received by the client. But since map iterator ordering can vary, this test may still succeed even without any patch.

Then the PR is doing a patch in the ServerTransport.disposeNow method in order to sort the channelGroup so we first handle Stream channels before handling parent ServerChannel.

Side note: when the server is configured only with H2 or with H2C, then only the stream channels (Http2MultiplexHandlerStreamChannel) are registered in the channelGroup, and the corresponding parent channels are not registered, so we don't have the problem.
However, it may be a problem to not have the parent channels registered, because in this case, if no active streams are currently being processed, then no GOAWAY message will be sent at all, which may be a problem, because if I'm correct we should always send a GOAWAY. But this is another use case, which might be addressed in another PR, if necessary.

Related to #2735

[pderop]

In the last commit, the impact of the patch is now minimal, we only take care to sort the channelGroup in order to ensure that streams channel are handled first, before any parent server channels.

[pderop]

@violetagg , can you take a look ?

[pderop]

@violetagg ,

can you retake a look, I just fixed a typo, and the channel group list is not sorted anymore.

[pderop]

@violetagg , thanks.