.Net 6 RC1 - Cannot set reponse headers in ExceptionMiddleware returning IAsyncEnumerable in controller

[gabynevada]

Describe the bug

This is probably related to the prior issue: #35364

In .Net 6 RC1 when IAsyncEnumerable is cancelled or fails, I have an exception middleware that transforms the error into a ProblemDetails json object to send to the client. When trying to set the response headers I'm having an exception "System.InvalidOperationException: Headers are read-only, response has already started.".

The workaround provided in the prior issue does not work with middleware.

To Reproduce

Use this webapi minimal repository to reproduce

https://github.com/gabynevada/.net6-iasync-enumerable-set-header-error

Set up connection to SQL database (Do not create database, to trigger failure in requests)

var sqlServerConnectionString = "Server=localhost;Database=Application;user id=sa;password=MyP@assword;";

Call endpoint returning IAsyncEnumerable

api/clients/pipeline-error

Or endpoint utilizing posible solution discussed in the prior issue #35364

api/clients/possible-solution-with-pipeline-error

Exceptions (if any)

Click to view output

fail: Microsoft.EntityFrameworkCore.Database.Connection[20004] An error occurred using the connection to database 'Application' on server 'localhost'. fail: Microsoft.EntityFrameworkCore.Query[10100] An exception occurred while iterating over the results of a query for context type 'SetResponseHeaders.Db.ApplicationContext'. Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server) at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at Microsoft.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity, Boolean withFailover, SqlAuthenticationMethod authType) at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover) at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout) at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance) at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool) at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions) at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions) at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen() --- End of stack trace from previous location --- at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync() ClientConnectionId:00000000-0000-0000-0000-000000000000 Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server) at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at Microsoft.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity, Boolean withFailover, SqlAuthenticationMethod authType) at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover) at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout) at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance) at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool) at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions) at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions) at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen() --- End of stack trace from previous location --- at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync() ClientConnectionId:00000000-0000-0000-0000-000000000000 fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1] An unhandled exception has occurred while executing the request. System.InvalidOperationException: Headers are read-only, response has already started. at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpHeaders.ThrowHeadersReadOnlyException() at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseHeaders.Microsoft.AspNetCore.Http.IHeaderDictionary.set_ContentType(StringValues value) at Microsoft.AspNetCore.Http.DefaultHttpResponse.set_ContentType(String value) at SetResponseHeaders.ExceptionMiddleware.HandleException(HttpContext context, Exception error) in /Users/elvis/RiderProjects/.net6-iasync-enumerable-set-header-error/ExceptionMiddleware.cs:line 34 at SetResponseHeaders.ExceptionMiddleware.Invoke(HttpContext httpContext) in /Users/elvis/RiderProjects/.net6-iasync-enumerable-set-header-error/ExceptionMiddleware.cs:line 28 at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext) at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider) at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context) warn: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[2] The response has already started, the error page middleware will not be executed. fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1] An unhandled exception has occurred while executing the request. System.InvalidOperationException: Headers are read-only, response has already started. at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpHeaders.ThrowHeadersReadOnlyException() at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseHeaders.Microsoft.AspNetCore.Http.IHeaderDictionary.set_ContentType(StringValues value) at Microsoft.AspNetCore.Http.DefaultHttpResponse.set_ContentType(String value) at SetResponseHeaders.ExceptionMiddleware.HandleException(HttpContext context, Exception error) in /Users/elvis/RiderProjects/.net6-iasync-enumerable-set-header-error/ExceptionMiddleware.cs:line 34 at SetResponseHeaders.ExceptionMiddleware.Invoke(HttpContext httpContext) in /Users/elvis/RiderProjects/.net6-iasync-enumerable-set-header-error/ExceptionMiddleware.cs:line 28 at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext) at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider) at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context) warn: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[2] The response has already started, the error page middleware will not be executed. fail: Microsoft.AspNetCore.Server.Kestrel[13] Connection id "0HMC0BQ7N250E", Request id "0HMC0BQ7N250E:00000002": An unhandled exception was thrown by the application. System.InvalidOperationException: Headers are read-only, response has already started. at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpHeaders.ThrowHeadersReadOnlyException() at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseHeaders.Microsoft.AspNetCore.Http.IHeaderDictionary.set_ContentType(StringValues value) at Microsoft.AspNetCore.Http.DefaultHttpResponse.set_ContentType(String value) at SetResponseHeaders.ExceptionMiddleware.HandleException(HttpContext context, Exception error) in /Users/elvis/RiderProjects/.net6-iasync-enumerable-set-header-error/ExceptionMiddleware.cs:line 34 at SetResponseHeaders.ExceptionMiddleware.Invoke(HttpContext httpContext) in /Users/elvis/RiderProjects/.net6-iasync-enumerable-set-header-error/ExceptionMiddleware.cs:line 28 at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext) at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider) at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Further technical details

• ASP.NET Core version

6.0.100-rc.1.21458.32

• Include the output of dotnet --info

Click to view output

.NET SDK (reflecting any global.json): Version: 6.0.100-rc.1.21458.32 Commit: d7c22323c4

Runtime Environment:
OS Name: Mac OS X
OS Version: 11.6
OS Platform: Darwin
RID: osx.11.0-x64
Base Path: /usr/local/share/dotnet/sdk/6.0.100-rc.1.21458.32/

Host (useful for support):
Version: 6.0.0-rc.1.21451.13
Commit: d7619cd4b1

.NET SDKs installed:
5.0.100 [/usr/local/share/dotnet/sdk]
5.0.101 [/usr/local/share/dotnet/sdk]
5.0.102 [/usr/local/share/dotnet/sdk]
5.0.103 [/usr/local/share/dotnet/sdk]
5.0.202 [/usr/local/share/dotnet/sdk]
6.0.100-preview.7.21379.14 [/usr/local/share/dotnet/sdk]
6.0.100-rc.1.21458.32 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 5.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.3 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.5 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.0-preview.7.21378.6 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.0-rc.1.21452.15 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 5.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.1 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.3 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.0-preview.7.21377.19 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.0-rc.1.21451.13 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

- The IDE (VS / VS Code/ VS4Mac) you're running on, and its version

JetBrains Rider 2021.2.1

[davidfowl]

This is by design, you can't change the response if it's already started.

[pranavkm]

We discussed this in triage today. The issue is that SerializeAsync begins writing to the Stream before the first element has been iterated over. Here's a minimal repro for this:

var stream = new MemoryStream();
try
{
    await JsonSerializer.SerializeAsync(stream, AsyncEnumerable());
}
catch (Exception)
{
    Console.WriteLine("Written bytes: " + stream.Length);
}

static async IAsyncEnumerable<int> AsyncEnumerable()
{
    await Task.Yield();
    throw new Exception();
    yield break;
}

It's writing the [ char before it iterates. Interestingly nothing is written if the exception is thrown without an await e.g. if the AsyncEnumerable looks like so:

static async IAsyncEnumerable<int> AsyncEnumerable()
{
    throw new Exception();
    yield break;
}

@eiriktsarpalis are we able to defer writing the array opening until the first element is read? In web apps, it gives the users an opportunity to recover the response and provide a meaningful error.

[gabynevada]

@pranavkm That would be great, the issue now is that we're getting a 200 OK response with the '[' character in the payload. This makes it difficult to handle the errors on the client.

It would also be useful if we could write to the headers before iterating the first element.

[adityamandaleeka]

Triage: We could wrap the AsyncEnumerable in another AsyncEnumerable and then read the first entry before we serialize. @BrennanConroy will try this out.

[davidfowl]

That doesn't feel right....

[pranavkm]

@davidfowl we were looking to read and buffer the first element so that we can catch this class of errors. A fairly common scenario for IAsyncEnumerable is to plumb a db result thru and it feels like we should do something to improve the error experience with it.

[davidfowl]

Why not fix it in the serializer?

[pranavkm]

I discussed the option with @eiriktsarpalis, but it sounded like it might be tricky to fix this in the serializer for 6.0.

[davidfowl]

I wouldn't want to see this behavior in asp.net specifically. Either the workaround should be written for everyone using the serializer or it should be fixed in the serializer. The same problem exists for HttpClient and we shouldn't hack it there either

[eiriktsarpalis]

For context, changing the converter so that the [ token is only inserted after the first element is fetched should serve to fix the particular scenario, however it would still fail in other cases (e.g. when the IAsyncEnumerable is nested in an envelope). Do note however that the current implementation uses a shared converter infrastructure for collection types so making IAsyncEnumerable behave slightly different would require a slight bit of churn. So I don't believe this is the correct approach...

The actual root cause turns out to be something different: the serializer maintains its own internal buffer and will only flush data to the underlying stream if at 90% capacity. The default buffer size is 16k and can be configured via the JsonSerializerOptions.DefaultBufferSize property. Note however that in the IAsyncEnumerable case the serializer will actually flush its internal buffer every time the async enumerator produces a pending task regardless of internal buffer capacity. This is by design, and is intended as a mechanism for the serializer to echo the chunking strategy employed by the source async enumerable.

One possible fix might be to disable forced flushes when the first MoveNextAsync() operation is pending. This would solve the issue in the general case (provided a reasonable JsonSerializerOptions.DefaultBufferSize setting has been provided). We would need to be careful about servicing this in 6.0 since it would need to add more logic to the core async serialization loop, which has historically been susceptible to performance regressions.

cc @steveharter @layomia

[eiriktsarpalis]

We just merged dotnet/runtime#59865 so a fix should be available in GA.

[BrennanConroy]

Confirmed the issue is resolved on main, and should be good when GA bits reach us.

[davidfowl]

Thanks for the fix @eiriktsarpalis !