Exception from RabbitMQ is treated as "user-unhandled" and stops Aspire app execution

[karolz-ms]

Repro steps

1. Install Visual Studio public preview (used VS Enterprise 2022 Preview 17.11.0 Preview 2.1, which is the latest at the time of install).
2. Make sure "Just My Code" is turned on.
3. Open Aspire solution (in the Aspire main repository) and set TestShop (in the "playground" folder) as startup project
4. F5

Expected

The app should start normally

Actual

The following exception is reported as "user-unhandled" and the execution stops

RabbitMQ.Client.Exceptions.BrokerUnreachableException
  HResult=0x80131620
  Message=None of the specified endpoints were reachable
  Source=RabbitMQ.Client
  StackTrace:
   at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
   at RabbitMQ.Client.ConnectionFactory.CreateConnection(String clientProvidedName)
   at RabbitMQ.Client.ConnectionFactory.CreateConnection()
   at Microsoft.Extensions.Hosting.AspireRabbitMQExtensions.<>c.<CreateConnection>b__7_0(IConnectionFactory factory) in C:\Users\karolz\code\aspire\src\Components\Aspire.RabbitMQ.Client\AspireRabbitMQExtensions.cs:line 188
   at Polly.ResiliencePipeline.<>c__36`2.<Execute>b__36_0(ResilienceContext _, ValueTuple`2 state)
 
  This exception was originally thrown at this call stack:
    RabbitMQ.Client.Impl.InboundFrame.ReadFrom(System.IO.Stream, byte[], System.Buffers.ArrayPool<byte>, uint)
    RabbitMQ.Client.Impl.SocketFrameHandler.ReadFrame()
    RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()
    RabbitMQ.Client.Framing.Impl.Connection.MainLoop()
 
Inner Exception 1:
IOException: connection.start was never received, likely due to a network timeout
 
Inner Exception 2:
EndOfStreamException: Reached the end of the stream. Possible authentication failure.

Additional info

This does NOT repro (execution continues normally) if Just My Code is turned off

FYI @DamianEdwards

[DamianEdwards]

This has been this way since the very beginning right? I was talking to @eerhardt about this just recently. The RabbitMQ client causes the debugger to break on every failed connection attempt until RabbitMQ is ready. Would be great to understand why though, as when we looked at this originally it wasn't obvious what the cause was as the logic is in a retry loop with an exception handler.

[karolz-ms]

Possibly. I suspect the latest Visual Studio previews have been more aggressive in terms of re-setting Just My Code option to enabled, which is why I noticed this issue now.

That said, something seems wrong--the execution of the program seems to stop for no good reason. Either the RabbitMQ client should handle the exception better, or Visual Studio is unnecessarily stopping when this exception is thrown (when Just My Code=true).

[eerhardt]

I can reproduce this behavior outside of Aspire by F5'ing the following Console App and enabling "Just My Code":

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="RabbitMQ.Client" Version="6.8.1" />
    <PackageReference Include="Polly.Core" Version="8.4.1" />
  </ItemGroup>

</Project>

using Polly;
using Polly.Retry;
using RabbitMQ.Client;
using RabbitMQ.Client.Exceptions;
using System.Net.Sockets;

var factory = new ConnectionFactory();
factory.Uri = new Uri("amqp://guest:guest@localhost:5672");

CreateConnection(factory, 3);

static IConnection CreateConnection(IConnectionFactory factory, int retryCount)
{
    var resiliencePipelineBuilder = new ResiliencePipelineBuilder();
    if (retryCount > 0)
    {
        resiliencePipelineBuilder.AddRetry(new RetryStrategyOptions
        {
            ShouldHandle = static args => args.Outcome is { Exception: SocketException or BrokerUnreachableException }
                ? PredicateResult.True()
                : PredicateResult.False(),
            BackoffType = DelayBackoffType.Exponential,
            MaxRetryAttempts = retryCount,
            Delay = TimeSpan.FromSeconds(1),
        });
    }
    var resiliencePipeline = resiliencePipelineBuilder.Build();

    return resiliencePipeline.Execute(factory =>
    {
        try
        {
            return factory.CreateConnection();
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.ToString());
            throw;
        }
    }, factory);
}

@BillHiebert - can someone on the VS Debugger take a look here? I don't think there is anything we can do from the Aspire side.

One interesting side note is that the exception ToString() is getting written to the console before VS breaks. The app is stopped, at line return factory.CreateConnection(); according to VS, but the exception has already been written to the console.

[BillHiebert]

@gregg-miskelly Is this expected with Just My Code? With Erik's example, if I just run the app it crashes at the end since it throws out of the entry point and is expected. If I debug with just my code disable, the program runs and only breaks on the final unhandled exception. But with just my code enabled, it breaks on the external throw out of RabbitMQ - all 3 since Erik's example has 3 retries:

 	[Exception] RabbitMQ.Client.dll!RabbitMQ.Client.ConnectionFactory.CreateConnection(RabbitMQ.Client.IEndpointResolver endpointResolver, string clientProvidedName)	Unknown
 	[Exception] RabbitMQ.Client.dll!RabbitMQ.Client.ConnectionFactory.CreateConnection(string clientProvidedName)	Unknown
 	[Exception] RabbitMQ.Client.dll!RabbitMQ.Client.ConnectionFactory.CreateConnection()	Unknown
>	[Exception] ConsoleApp2.dll!Program.<Main>$.AnonymousMethod__0_1(RabbitMQ.Client.IConnectionFactory factory) Line 33	C#
 	[Exception] Polly.Core.dll!Polly.ResiliencePipeline.Execute.AnonymousMethod__36_0(Polly.ResilienceContext _, (System.Func<TState, TResult> callback, TState state) state)	Unknown

[gregg-miskelly]

This is currently "By Design" -- the exception is travelling through user code and presumably being caught by Poly (system code), so the debugger will stop. There is currently a .NET 9 API proposal that would potentially address this if Poly was updated to take advantage of it (and the proposal landed on something that would work for Poly) -- dotnet/runtime#103105

[karolz-ms]

Thanks, everyone, much appreciated. I guess I am going to close this issue, and maybe open a new one for Polly if/when dotnet/runtime#103105 lands.