System.Text.Json.Serialization.Tests.StreamTests_Async.HandleCollectionsAsync failing in runtime-extra-platforms

[runfoapp]

System.Text.Json.Serialization.Tests.StreamTests_Async.HandleCollectionsAsync [FAIL]
    System.Text.Json.JsonReaderException : 'V' is invalid after a value. Expected either ',', '}', or ']'. LineNumber: 12 | BytePositionInLine: 59005.
    Stack Trace:
    /_/src/libraries/System.Text.Json/src/System/Text/Json/ThrowHelper.cs(279,0): at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
    /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1711,0): at System.Text.Json.Utf8JsonReader.ConsumeNextToken(Byte marker)
    /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1702,0): at System.Text.Json.Utf8JsonReader.ConsumeNextTokenOrRollback(Byte marker)
    /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(884,0): at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
    /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(278,0): at System.Text.Json.Utf8JsonReader.Read()
    /_/src/libraries/System.Text.Json/src/System/Text/Json/Document/JsonDocument.cs(961,0): at System.Text.Json.JsonDocument.Parse(ReadOnlySpan`1 utf8JsonSpan, JsonReaderOptions readerOptions, MetadataDb& database, StackRowStack& stack)
    /_/src/libraries/System.Text.Json/src/System/Text/Json/Document/JsonDocument.Parse.cs(686,0): at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 utf8Json, JsonReaderOptions readerOptions, Byte[] extraRentedArrayPoolBytes, PooledByteBufferWriter extraPooledByteBufferWriter)
    /_/src/libraries/System.Text.Json/src/System/Text/Json/Document/JsonDocument.Parse.cs(252,0): at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 json, JsonDocumentOptions options)
    /_/src/libraries/System.Text.Json/src/System/Text/Json/Document/JsonDocument.Parse.cs(318,0): at System.Text.Json.JsonDocument.Parse(String json, JsonDocumentOptions options)
    /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(21,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(String expected, String actual)
    /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(110,0): at System.Text.Json.Serialization.Tests.StreamTests.TestDeserialization[TElement](Stream memoryStream, String expectedJson, Type type, JsonSerializerOptions options)
    /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(76,0): at System.Text.Json.Serialization.Tests.StreamTests.PerformSerialization[TElement](Object obj, Type type, JsonSerializerOptions options)
    /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(53,0): at System.Text.Json.Serialization.Tests.StreamTests.RunTestAsync[TElement]()
    /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(24,0): at System.Text.Json.Serialization.Tests.StreamTests.HandleCollectionsAsync()

Runfo Tracking Issue: system.text.json.serialization.tests.streamtests_async.handlecollectionsasync

Build	Definition	Kind	Run Name

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
0	0	0

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Runfo Creating Tracking Issue (data being generated)

Author:	runfoapp[bot]
Assignees:	-
Labels:	area-System.Text.Json, untriaged
Milestone:	-

[elinor-fung]

Failed in build 1599183
Passed in build 1597902

Diff: ebcd332...56fcb74

[elinor-fung]

This is hitting a lot on Server 2022 (same with #65053). We will need to update the main runtime pipeline to use a Server 2022 queue soon. @eiriktsarpalis can we get someone to investigate soon?

[eiriktsarpalis]

Related to #59678 and #35927. The same issue has been popping up in other platforms as well and we have already disabled the test for some of these. I noticed on Kusto that similar failures have been occurring in other queues:

let crashes =
TestResults
| join kind=inner WorkItems on WorkItemId
| join kind=inner Jobs on JobId
//| where Started >= datetime(2022-01-01)
| where Result == "Fail"
| where  Method  == "HandleCollectionsAsync";
let min_t = toscalar(crashes | summarize min(Started));
let max_t = toscalar(crashes | summarize max(Started));
crashes
| make-series num=count() on Started in range(min_t, max_t, 1d) by QueueName
| render timechart

But nothing as prominent as Server 2022. This particular test has been giving us trouble historically, but issues have been hard to diagnose because it's not really a unit test -- it's doing too many things. I think we should just disable it completely on Windows to unblock rollout of the new queue. Longer term we should break up this test into real unit tests for easier diagnosis. cc @layomia

[carlossanlop]

Could this be related to the memory corruption issue reported here? #59678

[jeffhandley]

We've not had any more hits on this. It was probably the memory corruption issue.