JSON flaky tests

[krwq]

We're seeing some random JSON tests failing. So far I've seen following types of issues:

• random Assert.Contains failing where string seem to actually contain value but yet assert fails - see my comment here for more details - most likely there are other CurrentCulture related APIs which could be affected
• MemoryStream ends up being corrupted, in the specific cases we've seen this it was related to async code paths, see last stack trace here - this is most likely some kind of memory corruption but it's pretty rare and hard to pinpoint
• random tests failing with incorrect serialization/deserialization issues - this is most likely due to JsonTypeInfos or other fields in the options being initialized in multiple threads. We've already fixed some of these but still some issues exist.

related:

• System.Text.Json failing some large file tests #59678
• Test failed at double initialization assert in System.Text.Json.Serialization.Metadata.JsonTypeInfo.InitializePropCache() #60962
• Assertion failed in System.Text.Json.JsonSerializer.LookupProperty #67816
• Assertion failure in System.Text.Json.Serialization.Converters.ObjectWithParameterizedConstructorConverter1.TryLookupConstructorParameter` #68584

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

We're seeing some random JSON tests failing. So far I've seen following types of issues:

• random Assert.Contains failing where string seem to actually contain value but yet assert fails - see my comment here for more details - most likely there are other CurrentCulture related APIs which could be affected
• MemoryStream ends up being corrupted, in the specific cases we've seen this it was related to async code paths, see last stack trace here - this is most likely some kind of memory corruption but it's pretty rare and hard to pinpoint
• random tests failing with incorrect serialization/deserialization issues - this is most likely due to JsonTypeInfos or other fields in the options being initialized in multiple threads. We've already fixed some of these but still some issues exist.

related:

• System.Text.Json failing some large file tests #59678
• Test failed at double initialization assert in System.Text.Json.Serialization.Metadata.JsonTypeInfo.InitializePropCache() #60962

Author:	krwq
Assignees:	-
Labels:	area-System.Text.Json, untriaged
Milestone:	-

[krwq]

Seen on PR, seems perhaps we somehow ended up without properties on JsonTypeInfo (or one of the flags preventing serialization to happen)

    System.Text.Json.SourceGeneration.Tests.ReferenceHandlerTests_Metadata_AsyncStream.ValueTypesShouldNotContainId [FAIL]
      Assert.Equal() Failure
                  ↓ (pos 2)
      Expected: [{}]
      Actual:   [{"Name":"Angela","Job":{"Title":"Software ···
                  ↑ (pos 2)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/Common/ReferenceHandlerTests/ReferenceHandlerTests.Serialize.cs(113,0): at System.Text.Json.Serialization.Tests.ReferenceHandlerTests.ValueTypesShouldNotContainId()
        --- End of stack trace from previous location ---
  Finished:    System.Text.Json.SourceGeneration.Roslyn4.0.Tests
=== TEST EXECUTION SUMMARY ===
   System.Text.Json.SourceGeneration.Roslyn4.0.Tests  Total: 2944, Errors: 0, Failed: 1, Skipped: 0, Time: 2.023s
/private/tmp/helix/working/AE250924/w/A2B9090B/e
----- end Fri Apr 8 06:24:21 PDT 2022 ----- exit code 1 ----------------------------------------------------------

[krwq]

I've had this specific Assert.Contains fail 3 times overnight (out of 4 failures total):

    System.Text.Json.SourceGeneration.Tests.SerializationWithPerTypeAttributeContextTests.SerializeObjectArray_WithCustomOptions [FAIL]
      Assert.Contains() Failure
      Not found: featuredCampaign
      In value:  [{"activeOrUpcomingEvents":[{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"}],"featuredCampaign":{"id":234235,"title":"Promoting Open Source","description":"Very nice campaign","imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","organizationName":"The Company XYZ","headline":"The Headline"},"isNewAccount":false,"hasFeaturedCampaign":true},{"id":234235,"title":"Promoting Open Source","description":"Very nice campaign","imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","organizationName":"The Company XYZ","headline":"The Headline"}]
      Stack Trace:
        D:\src\runtime-4\src\libraries\System.Text.Json\tests\System.Text.Json.SourceGeneration.Tests\SerializationContextTests.cs(337,0): at System.Text.Json.SourceGeneration.Tests.SerializationContextTests.SerializeObjectArray_WithCustomOptions()

[ViktorHofer]

Seen on PR, seems perhaps we somehow ended up without properties on JsonTypeInfo (or one of the flags preventing serialization to happen)

@krwq this happened in this rolling build as well: https://dnceng.visualstudio.com/public/_build/results?buildId=1712332&view=ms.vss-test-web.build-test-results-tab&runId=46549310&resultId=144498&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

[danmoseley]

Should we fix the ordering of expected and actual in the asserts meantime..

[krwq]

Removing blocking-clean-ci label because most of the flakiness is already fixed and (I think) only remaining issue is also marked with this label.

[krwq]

moving this to 8 since most of the issues are fixed and we likely won't be able to track the remainder in 7.0 timeline

[eiriktsarpalis]

I think we can close this under the assumption that the final metadata race conditions have been addressed by #71630. If any remaining test failures crop up we can address them individually.