Fail nicely for non-blittable function pointer signatures

[Perksey]

Overview

Today I spent too long debugging an error code of -1073740940 only to find that my source generator wasn't manually marshalling string returns for function pointers like it's supposed to. This is fine because (eventually) I got to where I was going wrong, however the manor in which is failed was less than desirable.

Actual Suggestion

To ensure noone experiences something like tonight's traumatic debugging experience again, I'd like to suggest that upon calling a function pointer, throw a managed exception if any of the parameters or return value is non-blittable such that proceeding with the call will result in a runtime failure, rather than continuing to a critical error.

Extra Information

Here's some information on what led me to make this suggestion and how the runtime currently deals with a failure of this sort.

Function pointer signature:

delegate* cdecl<string>

WinDbg stack trace:

# Child-SP          RetAddr               Call Site
00 000000bf`38dadb50 00007ffa`974b7ae2     ntdll!RtlReportCriticalFailure+0x56
01 000000bf`38dadc40 00007ffa`974b7dca     ntdll!RtlpHeapHandleError+0x12
02 000000bf`38dadc70 00007ffa`974bd7f1     ntdll!RtlpHpHeapHandleError+0x7a
03 000000bf`38dadca0 00007ffa`973c9593     ntdll!RtlpLogHeapFailure+0x45
04 000000bf`38dadcd0 00007ffa`973c5d21     ntdll!RtlpFreeHeapInternal+0x433
05 000000bf`38dadd90 00007ff9`d9eb4b99     ntdll!RtlFreeHeap+0x51
06 000000bf`38daddd0 00007ff9`7eab2c11     System_Private_CoreLib!System.StubHelpers.CSTRMarshaler.ClearNative(IntPtr)$##6002734+0x39
07 000000bf`38dadea0 00007ff9`7eab2bab     0x00007ff9`7eab2c11
08 000000bf`38dadf10 00007ff9`7eab2a8b     0x00007ff9`7eab2bab
09 000000bf`38dae010 00007ff9`7eab12ac     0x00007ff9`7eab2a8b
0a 000000bf`38dae070 00007ff9`7e869688     0x00007ff9`7eab12ac
0b 000000bf`38dae100 00007ff9`7e8693d5     0x00007ff9`7e869688
0c 000000bf`38dae190 00007ff9`7e86190d     0x00007ff9`7e8693d5
0d 000000bf`38dae2e0 00007ff9`de376b63     0x00007ff9`7e86190d
0e 000000bf`38dae4c0 00007ff9`de30cb62     coreclr!CallDescrWorkerInternal+0x83 [F:\workspace\_work\1\s\src\vm\amd64\CallDescrWorkerAMD64.asm @ 101] 
0f (Inline Function) --------`--------     coreclr!CallDescrWorkerWithHandler+0x57 [f:\workspace\_work\1\s\src\vm\callhelpers.cpp @ 70] 
10 000000bf`38dae500 00007ff9`de313b29     coreclr!MethodDescCallSite::CallTargetWorker+0x196 [f:\workspace\_work\1\s\src\vm\callhelpers.cpp @ 612] 
11 (Inline Function) --------`--------     coreclr!MethodDescCallSite::Call+0xb [f:\workspace\_work\1\s\src\vm\callhelpers.h @ 468] 
12 000000bf`38dae640 00007ff9`de313fb7     coreclr!RunMain+0x1f5 [f:\workspace\_work\1\s\src\vm\assembly.cpp @ 1558] 
13 000000bf`38dae820 00007ff9`de314761     coreclr!Assembly::ExecuteMainMethod+0x1cb [f:\workspace\_work\1\s\src\vm\assembly.cpp @ 1681] 
14 000000bf`38daebb0 00007ff9`de2321c1     coreclr!CorHost2::ExecuteAssembly+0x221 [f:\workspace\_work\1\s\src\vm\corhost.cpp @ 460] 
15 000000bf`38daed40 00007ffa`6ae64e2d     coreclr!coreclr_execute_assembly+0x101 [f:\workspace\_work\1\s\src\dlls\mscoree\unixinterface.cpp @ 412] 
16 000000bf`38daede0 00007ffa`6ae72e27     hostpolicy!coreclr_t::execute_assembly+0x2d [f:\workspace\_work\1\s\src\corehost\cli\hostpolicy\coreclr.cpp @ 152] 
17 000000bf`38daee30 00007ffa`6ae72a36     hostpolicy!run_app_for_context+0x387 [f:\workspace\_work\1\s\src\corehost\cli\hostpolicy\hostpolicy.cpp @ 247] 
18 000000bf`38daef90 00007ffa`6ae74262     hostpolicy!run_app+0x46 [f:\workspace\_work\1\s\src\corehost\cli\hostpolicy\hostpolicy.cpp @ 276] 
19 000000bf`38daefe0 00007ffa`6d333a9a     hostpolicy!corehost_main+0x132 [f:\workspace\_work\1\s\src\corehost\cli\hostpolicy\hostpolicy.cpp @ 390] 
1a 000000bf`38daf190 00007ffa`6d3373f4     hostfxr!hostfxr_close+0x76a
1b 000000bf`38daf280 00007ffa`6d335bfb     hostfxr!hostfxr_close+0x40c4
1c 000000bf`38daf370 00007ffa`6d3320d9     hostfxr!hostfxr_close+0x28cb
1d 000000bf`38daf4b0 00007ff7`7392fa1f     hostfxr!hostfxr_main_startupinfo+0x89
1e 000000bf`38daf5b0 00007ff7`7392fdf7     apphost!exe_start+0x63f [f:\workspace\_work\1\s\src\corehost\corehost.cpp @ 216] 
1f 000000bf`38daf7c0 00007ff7`73931fa8     apphost!wmain+0xc7 [f:\workspace\_work\1\s\src\corehost\corehost.cpp @ 289] 
20 (Inline Function) --------`--------     apphost!invoke_main+0x22 [d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 90] 
21 000000bf`38daf8e0 00007ffa`96ca6fd4     apphost!__scrt_common_main_seh+0x10c [d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288] 
22 000000bf`38daf920 00007ffa`973fcec1     KERNEL32!BaseThreadInitThunk+0x14
23 000000bf`38daf950 00000000`00000000     ntdll!RtlUserThreadStart+0x21

CLR stack

        Child SP               IP Call Site
000000BF38DADE00 00007ffa974aed02 [InlinedCallFrame: 000000bf38dade00] 
000000BF38DADE00 00007ff9d9eb4b8c [InlinedCallFrame: 000000bf38dade00] 
000000BF38DADDD0 00007ff9d9eb4b8c System.StubHelpers.CSTRMarshaler.ClearNative(IntPtr) [/_/src/System.Private.CoreLib/src/System/StubHelpers.cs @ 115]
000000BF38DADEA0 00007ff97eab2c11 ILStubClass.IL_STUB_PInvoke()
000000BF38DADF48 00007ff97eab2bab [InlinedCallFrame: 000000bf38dadf48] 
000000BF38DADF10 00007ff97eab2bab ILStubClass.IL_STUB_PInvoke()
000000BF38DAE010 00007ff97eab2a8b Silk.NET.SDL.Sdl.GetPlatformS() [F:\_Silk.NET\Silk.NET\src\Windowing\Silk.NET.SDL\2c7e2760-8e89-4330-a6fd-236ee0e2d8ad_Silk.NET.SilkTouch.NativeApiGenerator_Sdl.58080534.gen.cs @ 6296]
000000BF38DAE070 00007ff97eab12ac Silk.NET.Windowing.Sdl.SdlPlatform.get_IsViewOnly() [F:\_Silk.NET\Silk.NET\src\Windowing\Silk.NET.Windowing.Sdl\SdlPlatform.cs @ 48]
000000BF38DAE100 00007ff97e869688 Silk.NET.Windowing.Window.get_IsViewOnly() [F:\_Silk.NET\Silk.NET\src\Windowing\Silk.NET.Windowing.Common\Window.cs @ 65]
000000BF38DAE190 00007ff97e8693d5 Silk.NET.Windowing.Window.Create(Silk.NET.Windowing.WindowOptions) [F:\_Silk.NET\Silk.NET\src\Windowing\Silk.NET.Windowing.Common\Window.cs @ 87]
000000BF38DAE2E0 00007ff97e86190d Tutorial.Program.Main(System.String[]) [F:\_Silk.NET\Silk.NET\examples\CSharp\Tutorial 2.2 - Camera\Program.cs @ 99]
000000BF38DAE6A8 00007ff9de376b63 [GCFrame: 000000bf38dae6a8] 
000000BF38DAEC40 00007ff9de376b63 [GCFrame: 000000bf38daec40] 

[ghost]

Tagging subscribers to this area: @vitek-karas, @swaroop-sridhar, @agocke
See info in area-owners.md if you want to be subscribed.

[jkotas]

calling a function pointer, throw a managed exception if any of the parameters or return value is non-blittable such that proceeding with the call will result in a runtime failure, rather than continuing to a critical error.

Unmanaged function pointers do support non-blittable arguments for backward compatibility. Unmanaged function pointers have existed in IL since .NET Framework 1.0. They have been used extensively by managed C++ and other non-C# code generators. Changing the runtime to throw on unmanaged function pointers with non-blittable argument would be significant breaking change.

The standard marshaling rules apply for calling unmanaged function pointers with non-blittable arguments. The example that was crashing for you would work just fine if the unmanaged method returned allocated string (it is the standard marshaling rule for string return value).

Maybe this can be an analyzer?

[jkoritzinsky]

With the DisableRuntimeMarshallingAttribute, function pointer invocations in an assembly with that attribute will not do marshalling and will throw exceptions if any marshalling would be required.

I think this covers the scenario enough and that we don't need an analyzer.