Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System.Runtime.Serialization.Formatters.Tests crashing with DOUBLY_LINKED_FL #44250

Closed
stephentoub opened this issue Nov 4, 2020 · 16 comments
Closed

System.Runtime.Serialization.Formatters.Tests crashing with DOUBLY_LINKED_FL #44250

stephentoub opened this issue Nov 4, 2020 · 16 comments
Assignees
@PeterSolMS
Labels
area-GC-coreclr bug
Milestone
6.0.0

Comments

@stephentoub
Copy link
Member 

  Discovering: System.Runtime.Serialization.Formatters.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Serialization.Formatters.Tests (found 77 of 78 test cases)
  Starting:    System.Runtime.Serialization.Formatters.Tests (parallel test collections = on, max threads = 12)
    System.Runtime.Serialization.Formatters.Tests.DisableBitTests.DisabledAlwaysInBrowser [SKIP]
      Condition(s) not met: "IsBinaryFormatterSuppressedOnThisPlatform"
./RunTests.sh: line 161:  9258 Segmentation fault: 11  (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Serialization.Formatters.Tests.runtimeconfig.json --depsfile System.Runtime.Serialization.Formatters.Tests.deps.json xunit.console.dll System.Runtime.Serialization.Formatters.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/private/tmp/helix/working/B1CC09DC/w/9AF908B5/e
----- end Wed Nov 4 09:37:32 EST 2020 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.

e.g.
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-44245-merge-141c32ed6af649beb6/System.Runtime.Serialization.Formatters.Tests/console.63352ae9.log?sv=2019-07-07&se=2020-11-24T14%3A36%3A09Z&sr=c&sp=rl&sig=UG98YL2ecYRMMS4pgU6pF65nJVaC9MjU62QvSzk3jGk%3D

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-44245-merge-141c32ed6af649beb6/System.Runtime.Serialization.Formatters.Tests/core.9258?sv=2019-07-07&se=2020-11-24T14%3A36%3A09Z&sr=c&sp=rl&sig=UG98YL2ecYRMMS4pgU6pF65nJVaC9MjU62QvSzk3jGk%3D
@jkotas jkotas added the os-mac-os-x macOS aka OSX label Nov 4, 2020
@stephentoub
Copy link
Member Author

Failing on other OSes as well, it turns out, both Linux and Windows. Got one with an assert failure:

C:\h\w\C01B0A79\w\B9EE0A16\e>"C:\h\w\C01B0A79\p\dotnet.exe" exec --runtimeconfig System.Runtime.Serialization.Formatters.Tests.runtimeconfig.json --depsfile System.Runtime.Serialization.Formatters.Tests.deps.json xunit.console.dll System.Runtime.Serialization.Formatters.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Runtime.Serialization.Formatters.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Serialization.Formatters.Tests (found 78 of 79 test cases)
  Starting:    System.Runtime.Serialization.Formatters.Tests (parallel test collections = on, max threads = 2)
    System.Runtime.Serialization.Formatters.Tests.DisableBitTests.DisabledAlwaysInBrowser [SKIP]
      Condition(s) not met: "IsBinaryFormatterSuppressedOnThisPlatform"
Fatal error. Internal CLR error. (0x80131506)
   at System.Runtime.CompilerServices.RuntimeHelpers.AllocateUninitializedClone(System.Object)
   at System.Object.MemberwiseClone()
   at System.DefaultBinder.SelectMethod(System.Reflection.BindingFlags, System.Reflection.MethodBase[], System.Type[], System.Reflection.ParameterModifier[])
   at System.RuntimeType.GetMethodImplCommon(System.String, Int32, System.Reflection.BindingFlags, System.Reflection.Binder, System.Reflection.CallingConventions, System.Type[], System.Reflection.ParameterModifier[])
   at System.RuntimeType.GetMethodImpl(System.String, System.Reflection.BindingFlags, System.Reflection.Binder, System.Reflection.CallingConventions, System.Type[], System.Reflection.ParameterModifier[])
   at System.Type.GetMethod(System.String, System.Type[])
   at System.Runtime.Serialization.Formatters.Tests.EqualityExtensions.CheckEquals(System.Object, System.Object, Boolean)
   at System.Runtime.Serialization.Formatters.Tests.EqualityExtensions.CheckSequenceEquals(System.Collections.IEnumerable, System.Collections.IEnumerable, Boolean)
   at System.Runtime.Serialization.Formatters.Tests.EqualityExtensions.CheckEquals(System.Object, System.Object, Boolean)
   at System.Runtime.Serialization.Formatters.Tests.BinaryFormatterTests.ValidateBasicObjectsRoundtrip(System.Object, System.Runtime.Serialization.Formatters.FormatterAssemblyStyle, System.Runtime.Serialization.Formatters.TypeFilterLevel, System.Runtime.Serialization.Formatters.FormatterTypeStyle)
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean, Boolean)
   at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].CallTestMethod(System.Object)
   at Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<<InvokeTestMethodAsync>b__1>d<System.__Canon> ByRef)
   at Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<InvokeTestMethodAsync>b__1()
   at Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<AggregateAsync>d__4 ByRef)
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(System.Func`1<System.Threading.Tasks.Task>)
   at Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9, xunit.core, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__9 ByRef)
   at Xunit.Sdk.ExceptionAggregator.RunAsync(System.Func`1<System.Threading.Tasks.Task>)
   at Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<InvokeTestMethodAsync>d__48<System.__Canon> ByRef)
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InvokeTestMethodAsync(System.Object)
   at Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<<RunAsync>b__47_0>d<System.__Canon> ByRef)
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<RunAsync>b__47_0()
   at Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.core, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__10`1<System.Decimal> ByRef)
   at Xunit.Sdk.ExceptionAggregator.RunAsync[[System.Decimal, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Func`1<System.Threading.Tasks.Task`1<System.Decimal>>)
   at Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<InvokeTestAsync>d__4 ByRef)
   at Xunit.Sdk.XunitTestRunner.InvokeTestAsync(Xunit.Sdk.ExceptionAggregator)
   at Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.core, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__10`1<System.__Canon> ByRef)
   at Xunit.Sdk.ExceptionAggregator.RunAsync[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Func`1<System.Threading.Tasks.Task`1<System.__Canon>>)
   at Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__43<System.__Canon> ByRef)
   at Xunit.Sdk.TestRunner`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync()
   at Xunit.Sdk.XunitTheoryTestCaseRunner+<RunTestAsync>d__11.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTheoryTestCaseRunner+<RunTestAsync>d__11, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestAsync>d__11 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.XunitTheoryTestCaseRunner+<RunTestAsync>d__11, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestAsync>d__11 ByRef)
   at Xunit.Sdk.XunitTheoryTestCaseRunner.RunTestAsync()
   at Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__19<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__19<System.__Canon> ByRef)
   at Xunit.Sdk.TestCaseRunner`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync()
   at Xunit.Sdk.XunitTheoryTestCase.RunAsync(Xunit.Abstractions.IMessageSink, Xunit.Sdk.IMessageBus, System.Object[], Xunit.Sdk.ExceptionAggregator, System.Threading.CancellationTokenSource)
   at Xunit.Sdk.XunitTestMethodRunner.RunTestCaseAsync(Xunit.Sdk.IXunitTestCase)
   at Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestCasesAsync>d__32<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestCasesAsync>d__32<System.__Canon> ByRef)
   at Xunit.Sdk.TestMethodRunner`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunTestCasesAsync()
   at Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__31<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__31<System.__Canon> ByRef)
   at Xunit.Sdk.TestMethodRunner`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync()
   at Xunit.Sdk.XunitTestClassRunner.RunTestMethodAsync(Xunit.Abstractions.ITestMethod, Xunit.Abstractions.IReflectionMethodInfo, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>, System.Object[])
   at Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestMethodsAsync>d__38<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestMethodsAsync>d__38<System.__Canon> ByRef)
   at Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunTestMethodsAsync()
   at Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__37<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__37<System.__Canon> ByRef)
   at Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync()
   at Xunit.Sdk.XunitTestCollectionRunner.RunTestClassAsync(Xunit.Abstractions.ITestClass, Xunit.Abstractions.IReflectionTypeInfo, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>)
   at Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestClassesAsync>d__28<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestClassesAsync>d__28<System.__Canon> ByRef)
   at Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunTestClassesAsync()
   at Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__27<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__27<System.__Canon> ByRef)
   at Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync()
   at Xunit.Sdk.XunitTestAssemblyRunner.RunTestCollectionAsync(Xunit.Sdk.IMessageBus, Xunit.Abstractions.ITestCollection, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>, System.Threading.CancellationTokenSource)
   at Xunit.Sdk.XunitTestAssemblyRunner+<>c__DisplayClass14_2.<RunTestCollectionsAsync>b__2()
   at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
   at System.Threading.Tasks.Task+<>c.<.cctor>b__278_0(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.Tasks.Task.ExecuteEntry()
   at System.Threading.Tasks.SynchronizationContextTaskScheduler+<>c.<.cctor>b__8_0(System.Object)
   at Xunit.Sdk.MaxConcurrencySyncContext.RunOnSyncContext(System.Threading.SendOrPostCallback, System.Object)
   at Xunit.Sdk.MaxConcurrencySyncContext+<>c__DisplayClass11_0.<WorkerThreadProc>b__0(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at Xunit.Sdk.MaxConcurrencySyncContext.WorkerThreadProc()
   at Xunit.Sdk.XunitWorkerThread+<>c.<QueueUserWorkItem>b__5_0(System.Object)
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task+<>c.<.cctor>b__278_0(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
   at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object)
   at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart(System.Object)

@stephentoub stephentoub removed the os-mac-os-x macOS aka OSX label Nov 4, 2020
@stephentoub
Copy link
Member Author

@jkotas, does this internal CLR error look familiar?

@jkotas
Copy link
Member

jkotas commented Nov 4, 2020

GC heap corruption:

 # Child-SP          RetAddr           Call Site
00 (Inline Function) --------`-------- coreclr!WKS::my_get_size+0xb [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 9702] 
01 000000b4`68e1b3f0 00007ffd`8539eb10 coreclr!WKS::gc_heap::mark_object_simple+0xa5 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 20259] 
02 (Inline Function) --------`-------- coreclr!WKS::gc_heap::mark_through_cards_helper+0x28 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 30791] 
03 000000b4`68e1b440 00007ffd`853b965b coreclr!WKS::gc_heap::mark_through_cards_for_uoh_objects+0x280 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 35547] 
04 000000b4`68e1b5a0 00007ffd`853b69ce coreclr!WKS::gc_heap::mark_phase+0x35b [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 21677] 
05 000000b4`68e1b650 00007ffd`8539c6b9 coreclr!WKS::gc_heap::gc1+0x96 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 17521] 
06 (Inline Function) --------`-------- coreclr!GCToOSInterface::GetLowPrecisionTimeStamp+0x5 [F:\workspace\_work\1\s\src\coreclr\src\vm\gcenv.os.cpp @ 1033] 
07 000000b4`68e1b6b0 00007ffd`85338ec0 coreclr!WKS::gc_heap::garbage_collect+0x515 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 19107] 
08 000000b4`68e1b700 00007ffd`8533897a coreclr!WKS::GCHeap::GarbageCollectGeneration+0x270 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 38893] 
09 000000b4`68e1b750 00007ffd`853dca8c coreclr!WKS::gc_heap::trigger_gc_for_alloc+0x32 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 14519] 
0a 000000b4`68e1b780 00007ffd`853dc7b1 coreclr!WKS::gc_heap::try_allocate_more_space+0x2b4 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 14637] 
0b 000000b4`68e1b7e0 00007ffd`8539ffc7 coreclr!WKS::gc_heap::allocate_more_space+0x31 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 15132] 
0c (Inline Function) --------`-------- coreclr!WKS::gc_heap::allocate+0x5a [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 15163] 
0d 000000b4`68e1b810 00007ffd`85325316 coreclr!WKS::GCHeap::Alloc+0x87 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 37892] 
0e 000000b4`68e1b840 00007ffd`85323edb coreclr!Alloc+0x9a [F:\workspace\_work\1\s\src\coreclr\src\vm\gchelpers.cpp @ 239] 
0f 000000b4`68e1b890 00007ffd`853235a0 coreclr!AllocateArrayEx+0x10b [F:\workspace\_work\1\s\src\coreclr\src\vm\gchelpers.cpp @ 657] 
10 000000b4`68e1b960 00007ffd`8535a6cf coreclr!DupArrayForCloning+0x90 [F:\workspace\_work\1\s\src\coreclr\src\vm\gchelpers.cpp @ 800] 
11 000000b4`68e1b9b0 00007ffd`26be3b81 coreclr!ObjectNative::AllocateUninitializedClone+0xff [F:\workspace\_work\1\s\src\coreclr\src\classlibnative\bcltype\objectnative.cpp @ 244] 
12 000000b4`68e1bb10 00000000`00000003 0x00007ffd`26be3b81

Looking for details

@stephentoub stephentoub changed the title System.Runtime.Serialization.Formatters.Tests crashing in CI on macOS System.Runtime.Serialization.Formatters.Tests crashing in CI Nov 4, 2020
@stephentoub stephentoub added the bug label Nov 4, 2020
@jkotas
Copy link
Member

jkotas commented Nov 4, 2020

ObjectHolder[] in the large object heap is corrupted.

0:010> !DumpArray /d 000000b46e541058
Name:        System.Runtime.Serialization.ObjectHolder[]
MethodTable: 00007ffd267e2528
EEClass:     00007ffd258767e0
Size:        8388632(0x800018) bytes
Array:       Rank 1, Number of elements 1048576, Type CLASS
Element Methodtable: 00007ffd267e2498
[0] null
[1] 000000b40106f780 // Invalid object
[2] null
....
[1048571] null
[1048572] 000000b40106f6c8 // Invalid object
[1048573] null
[1048574] 000000b40106f000 // Invalid object
[1048575] null

#43636 is likely suspect.

jkotas added a commit to jkotas/runtime that referenced this issue Nov 4, 2020
@jkotas
Disable DOUBLY_LINKED_FL
6d58275 
Fixes dotnet#44250
@jkotas
Copy link
Member

jkotas commented Nov 4, 2020

Trying revert in #44262

cc @Maoni0 @PeterSolMS

@Maoni0
Copy link
Member

Maoni0 commented Nov 4, 2020

I thought PR#43636 was clean but of course this could be due to timing so may only show up in some CI jobs. @PeterSolMS could you please take a look?

@ghost
Copy link

ghost commented Nov 4, 2020

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

@stephentoub stephentoub added this to the 6.0.0 milestone Nov 4, 2020
@jkotas
Copy link
Member

jkotas commented Nov 4, 2020 •
edited
Loading

If it helps, I have archived the dump of the crash at \\jkotas9\drops\issue-44250

@mangod9
Copy link
Member

mangod9 commented Nov 5, 2020

Assigning to Peter for now. Does reverting double-LL make things better, or is that just a theory?

@jkotas
Copy link
Member

jkotas commented Nov 5, 2020

The crash is hitting a lot of PRs. #44262 is running pretty well so far. If the crash does not show up in it, I am going to merge it and then submit another test PR to revert the revert to verify that it will make it crash again.

@mangod9
Copy link
Member

mangod9 commented Nov 5, 2020

ok cool, sounds good. Thx!

@jkotas jkotas closed this as completed in d291c6c Nov 5, 2020
@jkotas jkotas reopened this Nov 5, 2020
@jkotas
Copy link
Member

jkotas commented Nov 5, 2020

Revert of the revert: #44271

@jkotas
Copy link
Member

jkotas commented Nov 5, 2020

#44271 has the crashes back. I will keep the PR opened for a few days in case you would like to download dumps, etc.

@jkotas jkotas changed the title System.Runtime.Serialization.Formatters.Tests crashing in CI System.Runtime.Serialization.Formatters.Tests crashing with DOUBLY_LINKED_FL Nov 5, 2020
@PeterSolMS
Copy link
Contributor

@jkotas: I cannot access the dump at \jkotas9\drops\issue-44250. Will try to repro locally.

@PeterSolMS
Copy link
Contributor

I got a crash reproduced locally. Interestingly enough, my case is a little different, but also involves a huge array of object references.

@janvorli janvorli mentioned this issue Nov 5, 2020
Merged
@am11 am11 mentioned this issue Nov 5, 2020
Merged
@stephentoub stephentoub mentioned this issue Nov 7, 2020
Closed
@mangod9
Copy link
Member

mangod9 commented Nov 16, 2020

Fixed via #44585.

@mangod9 mangod9 closed this as completed Nov 16, 2020
@ghost ghost locked as resolved and limited conversation to collaborators Dec 16, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@PeterSolMS PeterSolMS

Labels
area-GC-coreclr bug
Projects
None yet
Milestone
6.0.0
Development

No branches or pull requests
5 participants
@stephentoub @jkotas @Maoni0 @PeterSolMS @mangod9