Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

76831-SystemNetQuicTestsQuicStreamTestsWriteCanceled_NextWriteThrows #92266

Conversation

rzikm
Copy link
Member

@rzikm rzikm commented Sep 19, 2023

Fixes #76831.

There was a data race between cancellation action firing (i.e. calling Abort()) and DisposeAsync() from the thread awaiting the cancelled Task.

Consider following code

                {
                    await using QuicStream stream = await connection.AcceptInboundStreamAsync();
                    CancellationTokenSource cts = new CancellationTokenSource(timeout);
                    //....
                    await stream.WriteAsync(buffer, cts.Token);
                }

If the cts times out, then the last await throws, which makes execution leave the block and triggering the DisposeAsync on stream. This dispose does graceful stream close. Which is racing with Abort() being called from the timer thread which handles the cts timeout.

This PR fixes the above situation by holding a lock while servicing the cancellation action (Abort), which mutually excludes DisposeAsync from gracefully closing the stream at the same time.

@ghost
Copy link

ghost commented Sep 19, 2023

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #76831.

There was a data race between cancellation action firing (i.e. calling Abort()) and DisposeAsync() from the thread awaiting the cancelled Task.

Consider following code

                {
                    await using QuicStream stream = await connection.AcceptInboundStreamAsync();
                    CancellationTokenSource cts = new CancellationTokenSource(timeout);
                    //....
                    await stream.WriteAsync(buffer, cts.Token);
                }

If the cts times out, then the last await throws, which makes execution leave the block and triggering the DisposeAsync on stream. This dispose does graceful stream close. Which is racing with Abort() being called from the timer thread which handles the cts timeout.

This PR fixes the above situation by holding a lock while servicing the cancellation action (Abort), which mutually excludes DisposeAsync from gracefully closing the stream at the same time.

Author: rzikm
Assignees: -
Labels:

area-System.Net.Quic

Milestone: -

@rzikm rzikm requested a review from ManickaP September 19, 2023 11:29
@rzikm rzikm marked this pull request as draft September 19, 2023 11:42
@ManickaP
Copy link
Member

I'm not sure this is still relevant after my last PR (#90253) in QuicStream, which reverted the order or calling abort and waking up the caller of the async operation:

CancellationAction = target =>
{
try
{
if (target is QuicStream stream)
{
stream.Abort(QuicAbortDirection.Read, stream._defaultErrorCode);
stream._receiveTcs.TrySetResult();
}
}
catch (ObjectDisposedException)
{
// We collided with a Dispose in another thread. This can happen
// when using CancellationTokenSource.CancelAfter.
// Ignore the exception
}
}

private readonly ResettableValueTaskSource _sendTcs = new ResettableValueTaskSource()
{
CancellationAction = target =>
{
try
{
if (target is QuicStream stream)
{
stream.Abort(QuicAbortDirection.Write, stream._defaultErrorCode);
}
}
catch (ObjectDisposedException)
{
// We collided with a Dispose in another thread. This can happen
// when using CancellationTokenSource.CancelAfter.
// Ignore the exception
}
}
};

For write, we now also wait for SEND_COMPLETE in all cases, including cancellation.

@rzikm
Copy link
Member Author

rzikm commented Sep 19, 2023

Closing, the bug was fixed by #90253

@rzikm rzikm closed this Sep 19, 2023
@jkotas jkotas deleted the 76831-SystemNetQuicTestsQuicStreamTestsWriteCanceled_NextWriteThrows branch September 21, 2023 17:20
@karelz karelz added this to the 9.0.0 milestone Oct 11, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Nov 10, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

System.Net.Quic.Tests.QuicStreamTests.WriteCanceled_NextWriteThrows test failure
3 participants