InvalidProgramException (invalid IL code), VerificationException, and SIGSEGV when executing LINQ queries simultaneously using thread pool

[cwrea]

Steps to Reproduce

1. Download, extract, and load the sample solution (zip file below) in a current version of Visual Studio. I tried with Visual Studio 15.6.4 for PC and Visual Studio 2017 for Mac version 7.5 Preview 1.

2. Build and run the Android app project, targeting a real Android device.

EFCoreBug5456.Android.zip

Expected Behavior

The sample app shouldn't crash when calling the test method during startup.

QueryBugsInMemoryTest.Repro5456_multi_level_include_group_join_is_per_query_context_async() is a test method from Entity Framework Core v2.0.2's test suite. This test method passes on both .NET Framework and .NET Core. I happened upon this issue as I've been assessing how a meaningful subset of EF Core's unit tests would perform with Xamarin.iOS and Xamarin.Android.

Actual Behavior

Most (90%) of the time, the app crashes. Crashes include exceptions System.InvalidProgramException (most frequent) and System.Security.VerificationException. One time the app was seen aborting with "Got a SIGSEGV while executing native code. This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application."

Examples of System.InvalidProgramException invalid IL in code include: method body is empty (most frequent), IL_0022: brtrue IL_00e9, IL_0021: ldelem.ref, IL_0083: brtrue IL_0088, IL_0028: callvirt 0x0000000b.

An example of System.Security.VerificationException is Invalid instruction target e9 occurred.

• Results reported when sample app run 10 times on a Moto G5 Plus

• Results reported when sample app run 10 times on a Google Nexus 7 (2013)

• Example of SIGSEGV abort with mono runtime stack trace

Of note is that the test executes in-memory LINQ queries simultaneousy using the thread pool via TPL's Parallel.For(). The location of these exceptions vary (especially in depth), but always within in a System.Object:(wrapper dynamic-method) object:lambda_method. The multiple locations, types of exceptions reported, and transient nature of the issue has me guessing there is corruption due to lack of thread safety in some part of IL code generation for lambda methods.

The issue has been observed occurring less frequently when lowering either the number of iterations performed by Parallel.For(), as well when limiting the maximum number of threads in the managed thread pool. The issue does not occur when employing a conventional for(), nor when limiting the Parallel.For() to a single thread with ParallelOptions { MaxDegreeOfParallelism = 1 }.

Potential Impact

While this is one issue from a broad test suite¹, the test originates from multiple cases of real world code from multiple developers, although no cases were in the context of a mobile application.

Yet it is highly conceivable that a mobile app could execute complex lamdba expressions simultaneously — especially as practices tend toward improving performance by taking advantage of multicore on slower power-efficient mobile CPUs.

A developer writing deliberately thread-safe code could still see failures due to this internal runtime issue.

¹ (and for the record, Xamarin.Android otherwise performs very well, as I'll share here soon-ish.)

Version Information

Microsoft Visual Studio Community 2017 
Version 15.6.4
VisualStudio.15.Release/15.6.4+27428.2015
Microsoft .NET Framework
Version 4.7.02556

Installed Version: Community

Visual C++ 2017   00369-60000-00001-AA398
Microsoft Visual C++ 2017

Visual F# Tools 10.1 for F# 4.1   00369-60000-00001-AA398
Microsoft Visual F# Tools 10.1 for F# 4.1

Add New File   3.5
The fastest and easiest way to add new files to any project - including files that start with a dot

Application Insights Tools for Visual Studio Package   8.11.10212.1
Application Insights Tools for Visual Studio

ASP.NET and Web Tools 2017   15.0.40314.0
ASP.NET and Web Tools 2017

ASP.NET Core Razor Language Services   1.0
Provides languages services for ASP.NET Core Razor.

ASP.NET Web Frameworks and Tools 2012   4.0.21208.0
For additional information, visit https://www.asp.net/

ASP.NET Web Frameworks and Tools 2017   5.2.51214.0
For additional information, visit https://www.asp.net/

Azure App Service Tools v3.0.0   15.0.40215.07/
Azure App Service Tools v3.0.0

C# Tools   2.7.0-beta3-62715-05. Commit Hash: db02128e6e3c4bdfc93e6ec425ac9162b4d4fe80
C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

CodeMaid   10.4.53
CodeMaid is an open source Visual Studio extension to cleanup and simplify our C#, C++, F#, VB, PHP, PowerShell, R, JSON, XAML, XML, ASP, HTML, CSS, LESS, SCSS, JavaScript and TypeScript coding.

Common Azure Tools   1.10
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

JavaScript Language Service   2.0
JavaScript Language Service

JavaScript Project System   2.0
JavaScript Project System

JavaScript UWP Project System   2.0
JavaScript UWP Project System

JetBrains ReSharper Ultimate 2017.2.2    Build 109.0.20171006.122324
JetBrains ReSharper Ultimate package for Microsoft Visual Studio. For more information about ReSharper Ultimate, visit http://www.jetbrains.com/resharper. Copyright © 2018 JetBrains, Inc.

Merq   1.1.17-rc (cba4571)
Command Bus, Event Stream and Async Manager for Visual Studio extensions.

Microsoft Continuous Delivery Tools for Visual Studio   0.3
Simplifying the configuration of continuous build integration and continuous build delivery from within the Visual Studio IDE.

Microsoft JVM Debugger   1.0
Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

Microsoft MI-Based Debugger   1.0
Provides support for connecting Visual Studio to MI compatible debuggers

Microsoft Visual C++ Wizards   1.0
Microsoft Visual C++ Wizards

Microsoft Visual Studio Tools for Containers   1.1
Develop, run, validate your ASP.NET Core applications in the target environment. F5 your application directly into a container with debugging, or CTRL + F5 to edit & refresh your app without having to rebuild the container.

Microsoft Visual Studio VC Package   1.0
Microsoft Visual Studio VC Package

Mono Debugging for Visual Studio   4.9.11-pre (71eb098)
Support for debugging Mono processes with Visual Studio.

Node.js Tools   1.4.11027.3
Adds support for developing and debugging Node.js apps in Visual Studio

NuGet Package Manager   4.6.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.

Open Command Line   2.1.179
Opens a command line at the root of the project. Support for all consoles such as CMD, PowerShell, Bash etc. Provides syntax highlighting, Intellisense and execution of .cmd and .bat files.

ProjectServicesPackage Extension   1.0
ProjectServicesPackage Visual Studio Extension Detailed Info

Search in Velocity by Silverlake Software LLC   0.7.1
Provides a command to search in Velocity the offline documentation and docset viewer for Windows

SQL Server Data Tools   15.1.61801.210
Microsoft SQL Server Data Tools

SQLite & SQL Server Compact Toolbox   4.7
SQLite & SQL Server Compact Toolbox adds scripting, import, export, rename, query execution and much more to SQL Server Compact & SQLite Data Connections.

Syntax Visualizer   1.0
An extension for visualizing Roslyn SyntaxTrees.

TypeScript Tools   15.6.20202.3
TypeScript Tools for Microsoft Visual Studio

Visual Basic Tools   2.7.0-beta3-62715-05. Commit Hash: db02128e6e3c4bdfc93e6ec425ac9162b4d4fe80
Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Visual Studio Code Debug Adapter Host Package   1.0
Interop layer for hosting Visual Studio Code debug adapters in Visual Studio

Visual Studio Tools for Universal Windows Apps   15.0.27428.01
The Visual Studio Tools for Universal Windows apps allow you to build a single universal app experience that can reach every device running Windows 10: phone, tablet, PC, and more. It includes the Microsoft Windows 10 Software Development Kit.

VisualStudio.Mac   1.0
Mac Extension for Visual Studio

VSColorOutput   2.5.1
Color output for build and debug windows - http://mike-ward.net/vscoloroutput

Xamarin   4.9.0.752 (0faa808ee)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin Designer   4.10.58 (cee1369d0)
Visual Studio extension to enable Xamarin Designer tools in Visual Studio.

Xamarin.Android SDK   8.2.0.16 (HEAD/a78295902)
Xamarin.Android Reference Assemblies and MSBuild support.

Xamarin.iOS and Xamarin.Mac SDK   11.8.1.28 (3979d08)
Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.

Log File

Here's output from adb logcat -d after a failed run: adb-logcat-d.txt

[cwrea]

This remains an issue on Visual Studio 15.7.0 with Xamarin 4.10.0.442 and Xamarin.Android SDK 8.3.0.19. Various crashes, as initially described above.

[rgroenewoudt]

I can confirm for VS 15.7.3 with various Android devices.

[cwrea]

@rgroenewoudt Thank you for confirming the issue, and also for trying on the latest Visual Studio release.

[Oleksandr-Tokmakov]

I have the same issue on VS 15.7.5 using EF Core extension methods. For example:

IList<Content> content = dbContext.Content.
                        AsNoTracking().
                        Include(nameof(Content.RelatedContent)).
                        ToList();

Causes ~ 1 crash out of 3 runs. More Include I do more often crash happens. 3 includes crashes app all the time on my galaxy note 8.

[rgroenewoudt]

VS 15.8, debug mode, EFCore 2.0.1 and 2.1.1

Run 1:

System.Security.VerificationException: Error in System.Object:(wrapper dynamic-method) object:lambda_method (System.Runtime.CompilerServices.Closure,Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor/TransparentIdentifier`2<Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor/TransparentIdentifier`2<Microsoft.EntityFrameworkCore.Query.QueryBugsInMemoryTest/Post5456, System.Collections.Generic.IEnumerable`1<Microsoft.EntityFrameworkCore.Query.QueryBugsInMemoryTest/Blog5456>>, Microsoft.EntityFrameworkCore.Query.QueryBugsInMemoryTest/Blog5456>) Invalid instruction 39

Run 2:

System.InvalidProgramException: Invalid IL code in (wrapper dynamic-method) object:lambda_method (System.Runtime.CompilerServices.Closure,Microsoft.EntityFrameworkCore.Query.QueryBugsInMemoryTest/Blog5456): method body is empty.

[Diraekt]

I can cofirm this. VS 15.8, EFCore 2.1.2

System.InvalidProgramException: Invalid IL code in (wrapper dynamic-method) object:lambda_method (System.Runtime.CompilerServices.Closure,Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor/TransparentIdentifier2<Microsoft.EntityFrameworkCore.Storage.ValueBuffer, Microsoft.EntityFrameworkCore.Storage.ValueBuffer>): method body is empty.`

at (wrapper managed-to-native) System.Delegate.CreateDelegate_internal(System.Type,object,System.Reflection.MethodInfo,bool)
at System.Delegate.CreateDelegate (System.Type type, System.Object firstArgument, System.Reflection.MethodInfo method, System.Boolean throwOnBindFailure, System.Boolean allowClosed) [0x002f0] in <43dbbdc147f2482093d8409abb04c233>:0
at System.Delegate.CreateDelegate (System.Type type, System.Object firstArgument, System.Reflection.MethodInfo method) [0x00000] in <43dbbdc147f2482093d8409abb04c233>:0
at System.Reflection.Emit.DynamicMethod.CreateDelegate (System.Type delegateType, System.Object target) [0x0001a] in <43dbbdc147f2482093d8409abb04c233>:0
at (wrapper dynamic-method) System.Object.lambda_method(System.Runtime.CompilerServices.Closure,CompanyContext.Entities.PersonEntity)
at System.Linq.Enumerable+SelectIPartitionIterator2[TSource,TResult].MoveNext () [0x00048] in <c71a230fbf1c468a803f281b7366d3b2>:0 at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider+ExceptionInterceptor1+EnumeratorExceptionInterceptor[T].MoveNext () [0x00034] in <159bf28779664ac6914c3caf595e15c9>:0

Any updates?

[rgroenewoudt]

Still reproducible with VS 15.9 and EFCore 2.1.4.

 Unhandled Exception:
 System.InvalidProgramException: Invalid IL code in (wrapper dynamic-method) object:lambda_method (System.Runtime.CompilerServices.Closure,Microsoft.EntityFrameworkCore.Query.QueryBugsInMemoryTest/Post5456): method body is empty.
 
   at (wrapper managed-to-native) System.Delegate.CreateDelegate_internal(System.Type,object,System.Reflection.MethodInfo,bool)
   at System.Delegate.CreateDelegate (System.Type type, System.Object firstArgument, System.Reflection.MethodInfo method, System.Boolean throwOnBindFailure, System.Boolean allowClosed) [0x002f0] in <c79446e93efd45a0b7bc2f9631593aff>:0 
   at System.Delegate.CreateDelegate (System.Type type, System.Object firstArgument, System.Reflection.MethodInfo method) [0x00000] in <c79446e93efd45a0b7bc2f9631593aff>:0 [0:] ..... parallel iteration #364 on thread with ManagedThreadId 8

   at System.Reflection.Emit.DynamicMethod.CreateDelegate (System.Type delegateType, System.Object target) [0x0001a] in <c79446e93efd45a0b7bc2f9631593aff>:0 
   at (wrapper dynamic-method) System.Object.lambda_method(System.Runtime.CompilerServices.Closure,Microsoft.EntityFrameworkCore.Query.QueryContext)
   at Microsoft.EntityFrameworkCore.InMemory.Storage.Internal.InMemoryDatabase+<>c__DisplayClass8_0`1[TResult].<CompileAsyncQuery>b__0 (Microsoft.EntityFrameworkCore.Query.QueryContext qc) [0x00000] in <6061297453c54990b767cc8ff4fd860f>:0 
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync[TResult] (System.Linq.Expressions.Expression query) [0x00036] in <9b2250d1fb7644b98af1ab5ed79e762f>:0 
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.ExecuteAsync[TResult] (System.Linq.Expressions.Expression expression) [0x00000] in <9b2250d1fb7644b98af1ab5ed79e762f>:0 
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1[TResult].System.Collections.Generic.IAsyncEnumerable<TResult>.GetEnumerator () [0x00011] in <9b2250d1fb7644b98af1ab5ed79e762f>:0 
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions+IncludableQueryable`2[TEntity,TProperty].System.Collections.Generic.IAsyncEnumerable<TEntity>.GetEnumerator () [0x00000] in <9b2250d1fb7644b98af1ab5ed79e762f>:0 
   at System.Linq.AsyncEnumerable.Aggregate_[TSource,TAccumulate,TResult] (System.Collections.Generic.IAsyncEnumerable`1[T] source, TAccumulate seed, System.Func`3[T1,T2,TResult] accumulator, System.Func`2[T,TResult] resultSelector, System.Threading.CancellationToken cancellationToken) [0x00016] in <afef21b57ad6402f8df4c5299ba699e2>:0 
   at Microsoft.EntityFrameworkCore.Query.QueryBugsInMemoryTest+<>c.<Repro5456_multi_level_include_group_join_is_per_query_context_async>b__2_0 (System.Int32 i) [0x0015e] in C:\Users\robbieg\Downloads\EFCoreBug5456.Android\EFCoreBug5456\QueryBugsInMemoryTest.cs:52 
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__6_1 (System.Object state) [0x00000] in <c79446e93efd45a0b7bc2f9631593aff>:0 
   at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context (System.Object state) [0x00007] in <c79446e93efd45a0b7bc2f9631593aff>:0 
   at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <c79446e93efd45a0b7bc2f9631593aff>:0 
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <c79446e93efd45a0b7bc2f9631593aff>:0 
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00021] in <c79446e93efd45a0b7bc2f9631593aff>:0 
   at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c79446e93efd45a0b7bc2f9631593aff>:0 
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c79446e93efd45a0b7bc2f9631593aff>:0 

[rgroenewoudt]

Still reproducible with VS 2019.1 and EFCore 2.2.4

[UKDeveloper99]

Same issue here, EF Core 2.2.4 and latest Xamarin.Android.

[UKDeveloper99]

Does anyone from Xamarin even look at these, open since 27 March 2018...?

[lambdageek]

Related issues on the Mono side: mono/mono#12626, possibly also mono/mono#14080. Also https://github.com/mono/mono/blob/d5b374bb51a07bf3525d63b4cef6ad6979c27d07/netcore/CoreFX.issues.rsp#L189-L193

[cwrea]

@lambdageek Thanks for connecting this to the other issues. Perhaps the repro here could be helpful as mono/mono#12626 mentions the issue as "unreproducible". The repro on this issue demonstrates the problems most of the time.

[marek-safar]

@lambdageek can we close this one?