Hedged Requests are not fired after connection timeouts

[sandrohanea]

Description

If a connection establishment is failing with a timeout, configured on the SocketsHttpHandler, hedged requests are not generated and the whole pipeline fails.

Reproduction Steps

PreReq:

• Have some standard hedging handler with 2 endpoint groups: [ "https://localhost:3000", "https://microsoft.com" ]
• Have total request timeout and endpoint timeout long enough to not be reached (e.g. 100 seconds)
• Have a short hedging delay.
• Configure a SocketsHttpHandler with ConnectTimeout short enough to be reached for the first endpoint group (e.g. 1 second)
• Fire a request with the newly created HttpClient

Expected behavior

The expected behavior is that "https://microsoft.com" endpoint is called and the pipeline succeeds.

Actual behavior

It is not actually called and a TaskCanceledException is thrown (coming from the HttpConnectionPool)

Task Canceled Exception
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> System.TimeoutException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.TimeoutException: A connection could not be established within the configured ConnectTimeout.
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnectionPool.CreateConnectTimeoutException(OperationCanceledException oce)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

[martintmk]

This code reproduces the issue:

using System;
using System.Net.Http;
using System.Net.Security;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http.Resilience;

var services = new ServiceCollection();
var clientBuilder = services
    .AddHttpClient("ClientR9")
    .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
    {
        UseProxy = false,
        Proxy = null,
        ConnectTimeout = TimeSpan.FromSeconds(1),
        SslOptions = new SslClientAuthenticationOptions
        {
            ClientCertificates = [],
            RemoteCertificateValidationCallback = (_, _, _, _) => true
        },
    })
    .AddStandardHedgingHandler(routing =>
    {
        routing.ConfigureOrderedGroups(g =>
        {
            g.Groups.Add(new UriEndpointGroup
            {
                Endpoints = new[] { new WeightedUriEndpoint { Uri = new Uri("https://localhost:3000") } }
            });

            g.Groups.Add(new UriEndpointGroup
            {
                Endpoints = new[] { new WeightedUriEndpoint { Uri = new Uri("https://microsoft.com") } }
            });
        });

    })
    .Configure(opt =>
    {
        opt.Hedging.MaxHedgedAttempts = 10;
        opt.Hedging.Delay = TimeSpan.FromSeconds(11);
        opt.Endpoint.CircuitBreaker.FailureRatio = 0.99;
        opt.Endpoint.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(900);
        opt.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(200);
        opt.Endpoint.Timeout.Timeout = TimeSpan.FromSeconds(200);
    });

var provider = services.BuildServiceProvider();
var clientFactory = provider.GetRequiredService<IHttpClientFactory>();
var client = clientFactory.CreateClient("ClientR9");

try
{
    var response = await client.GetAsync("https://localhost:3000");
    Console.WriteLine(response.StatusCode);
}
catch (OperationCanceledException ex)
{
    Console.WriteLine("Should never reach here but it does after 1 second");
}

The issue is that OperationCanceledException thrown from SocketsHttpHandler is not threated as transient. The challange is to detect this particular case and not treat other OperationCanceledException as transient.

[sandrohanea]

This is where the exception is thrown and unfortunately there is no consistent way in identifying if it was a task cancelled from the connection timeout or task cancelled from a request timeout, in which case we probably don't want to retry / execute hedged requests:

https://github.com/dotnet/runtime/blob/deb43368fde438b3e6928d3930f1362a4960fdf4/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs#L475

[martintmk]

How about checking against ResilienceContext.CancellationToken? If ResilienceContext.CancellationToken is not cancelled and we receive TaskCanceledException with that layout we know that this cancellation came from SocketsHttpHandler when establishing connection.

[sandrohanea]

I think that might work, what about the CancellationToken that is provided to the SendAsync method?
Is that used initially to create a linked CancellationToken for ResilienceContext.CancellationToken ?