CloseConnection and invocation cancellation / retry

[bernardnormier]

A connection sends a CloseConnection frame to its peer to initiate a graceful connection close (aka shutdown).

The CloseConnection frame is sent by initiateShutdown only when there is no outstanding upcall, for example:

ice/csharp/src/Ice/ConnectionI.cs

Line 1799 in fa22827

os.writeByte(Protocol.closeConnectionMsg);

ice/csharp/src/Ice/ConnectionI.cs

Line 1767 in fa22827

if (_state == StateClosing && _upcallCount == 0)

ice/csharp/src/Ice/ConnectionI.cs

Line 1094 in fa22827

if (_upcallCount == 0)

For upcallCount, see:

ice/csharp/src/Ice/ConnectionI.cs

Line 2949 in fa22827

// The upcall count keeps track of the number of dispatches, AMI (response) continuations, sent callbacks and

(it's the same in other languages)

upcallCount == 0 in particular guarantees there is no outstanding dispatch (since a dispatch is an upcall). This includes the dispatch of oneway requests.

Connection.close calls initiateShutdown only once there is no outstanding two-way invocations. See:

ice/csharp/src/Ice/ConnectionI.cs

Line 183 in fa22827

doApplicationClose();

ice/csharp/src/Ice/ConnectionI.cs

Line 475 in fa22827

doApplicationClose();

but other events (OA deactivation, Communicator destruction) don't check two-way invocations and will happily initiate shutdown (= send CloseConnection) with outstanding two-way invocations:

ice/csharp/src/Ice/ConnectionI.cs

Line 148 in fa22827

case ObjectAdapterDeactivated:

Now, this graceful connection closure is usually fast and successful, but it can sometimes take a while and even fail.

The typical reason for taking a while is the peer is busy with its own upcalls:

• dispatching two-way requests
• dispatching oneway requests
• dispatching two-way requests for invocations that have timed-out in the caller (= similar to oneway)

In the current code, we wait until the connection closure is complete to cancel two-way invocations in the peer (= the side that didn't send CloseConnection) that are outstanding when the peer receives the CloseConnection frame:

ice/csharp/src/Ice/ConnectionI.cs

Line 1245 in fa22827

foreach (OutgoingAsyncBase o in _asyncRequests.Values)

All these invocations are safe to cancel & retry since the CloseConnection frame guarantees none of these requests were dispatched (remember, the initiator waits until upcallCount is 0).

ice/csharp/src/Ice/Internal/OutgoingAsync.cs

Line 644 in fa22827

// A CloseConnectionException indicates graceful server shutdown, and is therefore

This wait delays the retrying of the invocations with a new connection (to the same server or another replica). Worse, these retry-able invocations may not be retried at all if the connection closure is not graceful for some reason (typically because it times out).

We should fix this behavior by canceling these invocations sooner--as soon as we receive the CloseConnection frame.

[bernardnormier]

In icerpc-csharp's implementation of the ice protocol, we do cancel outstanding invocations immediately when we receive a CloseConnection frame:

https://github.com/icerpc/icerpc-csharp/blob/071848c40365c3fe204f4e1d7866c22e4300dc1a/src/IceRpc/Internal/IceProtocolConnection.cs#L1201

https://github.com/icerpc/icerpc-csharp/blob/071848c40365c3fe204f4e1d7866c22e4300dc1a/src/IceRpc/IceRpcError.cs#L27

You need the retry interceptor to retry automatically on IceRpcError.InvocationCanceled:
https://docs.icerpc.dev/api/csharp/api/IceRpc.Retry.RetryInterceptor.html#IceRpc_Retry_RetryInterceptor_remarks

Without it, the caller gets the IceRpcError.InvocationCanceled error code and can then safely retry at the application level.