Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

R2R tests failing: Fatal error. Internal CLR error. (0x80131506) #44807

Closed
BruceForstall opened this issue Nov 17, 2020 · 8 comments
Closed

R2R tests failing: Fatal error. Internal CLR error. (0x80131506) #44807

BruceForstall opened this issue Nov 17, 2020 · 8 comments
Labels
area-ReadyToRun-coreclr blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs
Milestone
6.0.0

Comments

@BruceForstall
Copy link
Member

It appears that all R2R tests are failing, as follows:

:\h\w\A8B909AA\w\ACDF096F\e>dotnet C:\h\w\A8B909AA\p\xunit.console.dll JIT\CodeGenBringUpTests\JIT.CodeGenBringUpTests.XUnitWrapper.dll -parallel collections -nocolor -noshadow -xml testResults.xml -trait TestGroup=JIT.CodeGenBringUpTests 
Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (64-bit .NET 5.0.0)
  Discovering: JIT.CodeGenBringUpTests.XUnitWrapper
  Discovered:  JIT.CodeGenBringUpTests.XUnitWrapper
  Starting:    JIT.CodeGenBringUpTests.XUnitWrapper
Fatal error. Internal CLR error. (0x80131506)
   at System.Linq.Expressions.Expression.Call(System.Linq.Expressions.Expression, System.Reflection.MethodInfo, System.Collections.Generic.IEnumerable`1<System.Linq.Expressions.Expression>)
   at System.Linq.Expressions.Expression.Call(System.Reflection.MethodInfo, System.Linq.Expressions.Expression[])
   at Xunit.Sdk.ExecutionContextHelper.EnsureInitialized()
   at Xunit.Sdk.ExecutionContextHelper.get_IsSupported()
   at Xunit.Sdk.MaxConcurrencySyncContext.get_IsSupported()
   at Xunit.Sdk.XunitTestAssemblyRunner.SetupSyncContext(Int32)
   at Xunit.Sdk.XunitTestAssemblyRunner+<RunTestCollectionsAsync>d__14.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestAssemblyRunner+<RunTestCollectionsAsync>d__14, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestCollectionsAsync>d__14 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.XunitTestAssemblyRunner+<RunTestCollectionsAsync>d__14, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestCollectionsAsync>d__14 ByRef)
   at Xunit.Sdk.XunitTestAssemblyRunner.RunTestCollectionsAsync(Xunit.Sdk.IMessageBus, System.Threading.CancellationTokenSource)
   at Xunit.Sdk.TestAssemblyRunner`1+<RunAsync>d__41[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestAssemblyRunner`1+<RunAsync>d__41[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__41<System.__Canon> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestAssemblyRunner`1+<RunAsync>d__41[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__41<System.__Canon> ByRef)
   at Xunit.Sdk.TestAssemblyRunner`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync()
   at Xunit.Sdk.XunitTestFrameworkExecutor+<RunTestCases>d__8.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestFrameworkExecutor+<RunTestCases>d__8, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestCases>d__8 ByRef)
   at System.Runtime.CompilerServices.AsyncVoidMethodBuilder.Start[[Xunit.Sdk.XunitTestFrameworkExecutor+<RunTestCases>d__8, xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestCases>d__8 ByRef)
   at Xunit.Sdk.XunitTestFrameworkExecutor.RunTestCases(System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>, Xunit.Abstractions.IMessageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions)
   at Xunit.Sdk.TestFrameworkExecutor`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunTests(System.Collections.Generic.IEnumerable`1<Xunit.Abstractions.ITestCase>, Xunit.Abstractions.IMessageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions)
   at Xunit.Xunit2.RunTests(System.Collections.Generic.IEnumerable`1<Xunit.Abstractions.ITestCase>, Xunit.Abstractions.IMessageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions)
   at Xunit.ConsoleClient.ConsoleRunner.ExecuteAssembly(System.Object, Xunit.XunitProjectAssembly, Boolean, Boolean, System.Nullable`1<Boolean>, System.Nullable`1<Int32>, Boolean, Boolean, System.Nullable`1<Xunit.AppDomainSupport>, Boolean, Boolean, Xunit.XunitFilters, Boolean)
   at Xunit.ConsoleClient.ConsoleRunner.RunProject(Xunit.XunitProject, Boolean, System.Nullable`1<Boolean>, System.Nullable`1<Boolean>, System.Nullable`1<Int32>, Boolean, Boolean, System.Nullable`1<Xunit.AppDomainSupport>, Boolean, Boolean, Boolean)
   at Xunit.ConsoleClient.ConsoleRunner.EntryPoint(System.String[])
   at Xunit.ConsoleClient.Program.Main(System.String[])

This is for the R2R legs in the runtime-coreclr outerloop job, as well as the R2R specific jobs, runtime-coreclr r2r and runtime-coreclr r2r-extra. The crossgen2 and crossgen2-composite jobs are failing the same way.

https://dev.azure.com/dnceng/public/_build/results?buildId=888886&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=888602&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=887202&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=887278&view=results
https://dev.azure.com/dnceng/public/_build/results?buildId=887276&view=results

@dotnet/crossgen-contrib @mangod9
@BruceForstall BruceForstall added area-ReadyToRun-coreclr blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs labels Nov 17, 2020
@BruceForstall BruceForstall added this to the 6.0.0 milestone Nov 17, 2020
@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Nov 17, 2020
@runfoapp runfoapp bot mentioned this issue Nov 17, 2020
Closed
@nattress
Copy link
Contributor

These all started failing on 11/12 evening's scheduled runs. Could be related to #38457 since that was the only interesting infra change that day. I'll see if it repros locally.

@BruceForstall
Copy link
Member Author

cc @jkoritzinsky

@sandreenko
Copy link
Contributor

I was not able to repro it directly, the first failing run included these changes https://dev.azure.com/dnceng/public/_traceability/runview/changes?currentRunId=885251

@trylek
Copy link
Member

trylek commented Nov 17, 2020 •
edited
Loading

According to the AzDO pipeline log we're still using dotnet v5 as the CLI to run apps in Helix. I'm wondering whether it can be now causing problems if the tests claim to be targeting .NET 6. I'm also unable to repro locally and this is exactly the type of problem one can only see in Helix. I'll try to put up an instrumented run changing this to 6.

F:\workspace\_work\1\s\.dotnet\sdk\5.0.100-rc.2.20479.15\MSBuild.dll /nologo -maxcpucount /m /maxcpucount -verbosity:m /v:minimal /bl:F:\workspace\_work\1\s/artifacts/log/SendToHelix.binlog /clp:Summary /logger:F:\workspace\_work\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20552.5\tools\netcoreapp2.1\Microsoft.DotNet.Arcade.Sdk.dll /nr:False /p:ContinuousIntegrationBuild=True /p:TreatWarningsAsErrors=true /p:TargetArchitecture=x64 /p:TargetOS=windows /p:TargetOSSubgroup= /p:Configuration=Checked /warnaserror F:\workspace\_work\1\s/src/tests/Common/helixpublishwitharcade.proj
  DotNetCliVersion: 5.0.0
  DotNetCliPackageType: runtime

@jkoritzinsky
Copy link
Member

My PR didn't change anything in the CoreCLR build or test infra outside of some signing changes in the official build.

The failure seems to be happening in the CLI we pull down for testing, which is .NET 5.0 release as far as I can tell.

The crash is at https://github.com/dotnet/runtime/blob/release/5.0/src/coreclr/src/vm/methodtable.cpp#L5907

@sandreenko sandreenko mentioned this issue Nov 17, 2020
Merged
@nattress nattress mentioned this issue Nov 19, 2020
Merged
@nattress
Copy link
Contributor

David noticed we have multiple copies of some framework assemblies loaded in the crash dump. Running xunit.console from coreroot (which contains a development runtime and libraries) with a shared dotnet runtime is not a good idea. I created a PR which duplicates the xunit libraries from the runtime targetingpack into their own directory so random bits of the test runtime don't get loaded in the test harness itself.

@mangod9 mangod9 removed the untriaged New issue has not been triaged by the area owner label Nov 19, 2020
@nattress
Copy link
Contributor

The R2R tests are passing now in recent runtime outerloop runs.

@trylek
Copy link
Member

trylek commented Nov 20, 2020

Thanks Simon for fixing this so quickly!

@ghost ghost locked as resolved and limited conversation to collaborators Dec 20, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-ReadyToRun-coreclr blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs
Projects
None yet
Milestone
6.0.0
Development

No branches or pull requests
7 participants
@nattress @jkoritzinsky @BruceForstall @trylek @sandreenko @Dotnet-GitSync-Bot @mangod9