-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
configurable HTTP/2 PING timeouts in HttpClient #31198
Comments
Adding an HttpClient API to send pings on demand would be really complicated, pings are per connection and HttpClient doesn't expose any connection level APIs. Using a timer to send pings would be much easier. WebSockets has a similar feature already. |
Can you give some more detail on your scenario? Once you determine if a connection is still open, what actions will you be taking? |
@scalablecory As I imagine it, a response to a PING should not imply any action at all. This is just normal operating. I consider the things just in the opposit way: if the PING does not get a response, then the connection should be closed (an by getting an exception?!) |
Even the act of sending a ping would likely trigger an immediate exception for local disconnect scenarios. Not so much if there's a proxy involved. The API for a full round trip ping and ack would be tricky, especially since you could have multiple outstanding pings. WinHttp WebSockets has a feature where it will periodically send unsolicited ping acks (pongs) just to check if the connection is open. It starts with acks because it doesn't want a response, it's just trying to detect socket errors. |
I don't see how this would be a real world a scenario having multiple outstanding pings, at a given moment. A ping would be sent on a idle connection only. And there's no reason to send a second one before the first one was ACKed. Maybe there should be 2 parameters:
|
If it were an internal feature based on an idle timer then yes, but if Ping is a callable API then none of that is guaranteed. |
@chrisdot Thanks. Would you be using this as a way to minimize latency noticed by the user by opening alternative routes or offline alternatives prior to the next request? Your app would still need to handle the case of a request failing, as even if PING is successful 1 second earlier, the connection could have been killed in the mean time. But if used as the above, I can see some usefulness here.
Indeed, I don't see this as being a particularly difficult thing to implement. The one hiccup is that the API would only work for HTTP/2 (and in future, HTTP/3) connections. I don't see that as being a blocker, but I believe it'd be a first for |
As an concrete example of HTTP2 PING frames, consider gRPC--which was the impetus for @chrisdot opening this issue. The gRPC HTTP2 protocol explicitly uses PING frames:
The https://github.com/grpc/grpc-dotnet/ project is building a gRPC implementation completely in .NET. One of its goals is to be able to interoperate with the other gRPC implementations. The C core library implementation of gRPC implements the PING part of the protocol and calls it keepalive. The Go implementation is similar. |
@Tratcher : I think we don't need a direct method/API, a configurable timer manner (property) should be enough I guess. In my case I don't see the point of being able to send at the application level the PINGs. I think this is a transport level stuff and if it is transparent but configurable should be enough??
@scalablecory : yes, that's it! I would be able to know that the active connection is down even if on the application level I have no payload for a given time. And that's how it is used in gRPC in the streaming call model... (where everything started from)
@chwarr: this is exactly the root of my request: I would need to be able to set the pendant of the following ping parameters (taken from the grpc-core) to the grpc-dotnet:
|
@JamesNK do you think this will be needed for gRPC? Did you hear other requests? Triage: We could implement just timeout (based on fraction of existing timeouts like |
I don't know how difficult that is, and I don't fully understand your proposition based on fraction of existing timers. Sorry to insist :-) , but I think there is a tiny hole in the HTTP/2 implementation, which would make the streaming capability of gRPC quite unusable, and not interoperable with other stacks. But I guess there might be other applications over HTTP/2 that would make usage of the keep alive? (can't figure out what for the moment...) |
As @chrisdot have mentioned, this feature is critical for gRPC long lived RPCs and streaming RPCs. |
We need this functionality as well, but both in client and server .NET implementation. But now I wonder who will respond to client http2 pings on the server? We use Kestrel there. We need quick fix please, otherwise we are forced to use c server gRPC implementation. |
This is where my other related issue comes in... |
What I propose be added to public class SocketsHttpHandler
{
public TimeSpan KeepAliveInterval { get; set; }
public TimeSpan KeepAliveTimeout { get; set; }
public bool KeepAliveWithoutRequests { get; set; }
}
These are the most important settings from how gRPC clients generally handle keep alive: https://github.com/grpc/grpc/blob/master/doc/keepalive.md. There are additional settings for finer grain control of pings (some are server specific). I don't believe they are needed. |
The equivalent settings in the gRPC golang client: type ClientParameters struct {
// After a duration of this time if the client doesn't see any activity it
// pings the server to see if the transport is still alive.
// If set below 10s, a minimum value of 10s will be used instead.
Time time.Duration // The current default value is infinity.
// After having pinged for keepalive check, the client waits for a duration
// of Timeout and if no activity is seen even after that the connection is
// closed.
Timeout time.Duration // The current default value is 20 seconds.
// If true, client sends keepalive pings even with no active RPCs. If false,
// when there are no active RPCs, Time and Timeout will be ignored and no
// keepalive pings will be sent.
PermitWithoutStream bool // false by default.
} And gRPC Java client has keepAliveTime, keepAliveTimeout, keepAliveWithoutCalls:
|
Given that the properties impact only HTTP/2, I wonder if we should modify the names as well ... I was confused by the name |
Marking it for 5.0 for now due to raising demand ... |
QUIC has pings as well. We might get away with leaving out the @chrisdot, @JamesNK's proposal is not the more explicit 'is the connection alive right now' API you're asking for. Will keepalive be sufficient, or do you feel some sort of |
In my case I don't see the point in having a direct API to a ping itself... The need is just about checking the idle connections... |
That can be done with TCP keep-alive as well. (once the dialers are implemented) |
Naively, I would expect that Ping does not count as "normal" request and does not prolong idle timeout. User should set longer |
Ok. So for someone who wants to keep the connection "warm" they would combine idle timeout with keepalive. var socketsHttpHandler = new SocketsHttpHandler();
socketsHttpHandler.PooledConnectionIdleTimeout = TimeSpan.FromHours(2);
socketsHttpHandler.KeepAlivePingDelay = TimeSpan.FromSeconds(60);
socketsHttpHandler.KeepAlivePingTimeout = TimeSpan.FromSeconds(30);
socketsHttpHandler.EnableMultipleHttp2Connections = true; Does this scenario make sense:
etc I think this is how it will work, and it is probably fine. |
On
@Tratcher Have you looked closely at how Kestrel handles HTTP/2 pings compared to other servers? Update - I've looked up what gRPC servers do below. |
By default gRPC client implementations don't send keep alive pings without active streams. And by default the servers (CCore based gRPC servers, Java+Netty, golang) will close connections with GO_AWAY and ENHANCE_YOUR_CALM if the client attempts to ping the server without active streams. Pinging without active calls is different behavior from all other gRPC clients. Servers all have an option to allow it, but I think the difference behavior unnecessarily creates problems for .NET developers who want to call non-.NET gRPC servers and trying to get keep alive working. I think we should add |
Wow, that is an interesting design choice. Okay, we do need this to be configurable then. Thanks for providing the data! |
namespace System.Net.Http
{
public class SocketsHttpHandler
{
public TimeSpan KeepAlivePingDelay { get; set; }
public TimeSpan KeepAlivePingTimeout { get; set; }
}
} |
Yes it is desirable. I explained why it is desirable here - #31198 (comment) To go into an example: Imagine a client that infrequently sends bursts of data, e.g. 1000 HTTP requests once every hour. Without an option to send keep alive pings - even though there are no active calls - then a load balancer/proxy closes the connection for inactivity. Every time that client wakes up to send the 1000 HTTP requests, all have high latency while they wait for TCP connections to be created, TLS to be established, HTTP/2 connection to be established, and calls to be made. Customers - including Azure 1st party (I don't know if I can mention names publicly) - have asked for this feature. They have observed that the scenario impacts their @chwarr Also goes into detail about why This isn't gRPC specific. Anyone using HTTP/2 could use this. We're going around in circles. Could you please invite me to discuss this feature. |
Thanks @JamesNK. The function of them was never under question -- pushback we got in API review was on specific scenarios. P90 latency is a good scenario. |
Excellent. Let's approve this over email. |
namespace System.Net.Http
{
public class SocketsHttpHandler
{
public TimeSpan KeepAlivePingDelay { get; set; }
public TimeSpan KeepAlivePingTimeout { get; set; }
public HttpKeepAlivePingPolicy KeepAlivePingPolicy { get; set; }
}
public enum HttpKeepAlivePingPolicy
{
WithActiveRequests,
Always
}
} |
Justification
Long-running but idle connections (which happen in gRPC and long poll scenarios) can result in firewalls/etc. dropping connections out from under us, without any reset notification. This causes long timeouts on a client that is only reading data.
These APIs will have HttpClient periodically send ping frames over HTTP/2, which would prevent firewalls from seeing the connection as idle and detect otherwise broken connections sooner than we can right now.
API Proposal
Original issue below
I've been evaluating the new gRPC-dotnet and particularly its streaming feature in a long lived connected scenario.
I needed the HTTP/2 PING feature to detect the zombie connections (basically to know: "Am I still having an up and running connection to the client?") on both server and client side.
It seems that this feature won't be implement-able on the client side as long as there's no support of an API in the HttpClient, as the client part of gRPC-dotnet relies on HttpClient.
It seems that HttpClient already responds to ping but can not send any on demand. Am I right?
So I would like to request a new method in HttpClient class to support sending Http/2 pings or even better a kind of keep alive ping timeout to do it automatically in case there's no traffic...
BTW: I have also created an issue for the same feature for kestrel (I don't know what kestrel is using underneath).
The text was updated successfully, but these errors were encountered: