Crashes during debugging when pausing in VS2022/.NET9 #110552

kg · 2024-12-09T22:25:31Z

Description

Since upgrading to .NET 9, I get frequent debugger crashes when pausing in both Debug and Release configurations for my dictionary container. I haven't been able to notice any particular pattern to it.

Reproduction Steps

Run the 'Benchmarks' project from https://github.com/kg/SimdDictionary/ and hit Pause in VS during the self-test period while it is testing TailCollider, then make sure the Autos or Locals tab(s) are visible in the debugger. Local evaluation will fail and the application process will collapse if VectorizedDictionary code is currently running. Pausing in Main works though. This is not 100% reproducible - setting a breakpoint in a specific spot doesn't reproduce it every time - but if I run the self test and pause repeatedly it will eventually crash while evaluating locals. It seems to happen even if the autos/locals tabs aren't visible.

Expected behavior

No application crash even if local evaluation fails.

Actual behavior

Local evaluation fails and the application crashes.

---------------------------
Microsoft Visual Studio
---------------------------
The target process exited with code -1073740791 (0xC0000409) while evaluating the function ''.

If the problem happens regularly, consider disabling the Tools->Options setting "Debugging->General->Enable property evaluation and other implicit function calls" or debugging the cause by evaluating the expression from the Immediate window. See help for information on doing this.
---------------------------
OK   Help   
---------------------------

Regression?

I was able to get debugger crashes on .NET8 but it seems much much worse on 9. Could be a regression from upgrading VS, I'm not sure.

Known Workarounds

It looks like this could be related to some locals being NullRefs, so it's possible the workaround is to not use ref and Unsafe.NullRef at all, but that doesn't seem particularly reasonable.

Configuration

Visual Studio 2022 Version 17.12.3

PS C:\Users\kg> dotnet --list-runtimes
Microsoft.AspNetCore.App 8.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

x64 on a Ryzen 7950X

Other information

Sometimes it says evaluation failed for 'pair.Value' (but I don't have a watch on that, so what gives?)
Pair is defined like so:

[StructLayout(LayoutKind.Sequential, Pack = 1)]
internal struct Pair {
    public K Key;
    public V Value;
}

The text was updated successfully, but these errors were encountered:

dotnet-policy-service · 2024-12-09T22:25:56Z

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

kg · 2024-12-09T22:33:34Z

I'm happy to investigate this further locally if someone can give me tips on how to troubleshoot it. I'm kind of stumped because the process just goes away, I don't know how to get it into a mode where I can see what code is crashing during evaluation or where.

EDIT: I tried native debugging and it's worse, there's not even a popup, just this in the log:

'Benchmarks.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\9.0.0\System.Threading.Thread.dll'. Symbol loading disabled by Include/Exclude setting.
Unhandled exception at 0x00007FFFC0974120 (ntdll.dll) in Benchmarks.exe: Unknown __fastfail() status code: 0x0000000000000030.
The Common Language Runtime cannot stop at this exception. Common causes include: incorrect COM interop marshalling and memory corruption. To investigate further use native-only debugging.
The program '[62272] Benchmarks.exe' has exited with code 3221226505 (0xc0000409).

EDIT 2: The suggested "native-only debugging" does not seem to be an option in VS2022's UI anymore :(

EDIT 3: This suggests it's related to Vector128:

---------------------------
Microsoft Visual Studio
---------------------------
The target process exited with code -1073740791 (0xC0000409) while evaluating the function 'System.Runtime.Intrinsics.Vector128DebugView<T>.Vector128DebugView'.

If the problem happens regularly, consider disabling the Tools->Options setting "Debugging->General->Enable property evaluation and other implicit function calls" or debugging the cause by evaluating the expression from the Immediate window. See help for information on doing this.
---------------------------
OK   Help   
---------------------------

In retrospect, I've had problems with the debugger displaying incorrect values for Vector128s in the past while working on this, so it might be the problem.

tommcdon · 2024-12-10T00:41:04Z

Thanks for the bug report and investigation @kg! I was not able to reproduce on my machine (Intel I7), and so it is possible that it doesn't have all of the vectorization support needed to reproduce the bug.

As for investigating, could you try setting a conditional breakpoint in ShimProcess::HandleWin32DebugEvent, located in coreclr\debug\di\shimprocess.cpp, on the "IsExceptionEvent" line, with the following condition: pEvent->dwDebugEventCode == 1 && pEvent->u.Exception.ExceptionRecord.ExceptionCode == 0xc0000409? Then non-invasively attach windbg to the debuggee and dump the callstack (e.g. with the k command).

kg · 2024-12-10T00:57:28Z

Do I need to set that breakpoint in the debuggee or inside of VS? ShimProcess::HandleWin32DebugEvent's not resolving even with symbols loaded. I tried attaching windbg to msvsmon to see if I could set a BP on ShimProcess there and that didn't work either.

EDIT: From digging around by using a custom #include in a source server file, di\shimprocess.cpp isn't part of the symbols

tommcdon · 2024-12-10T01:08:58Z

@kg - to stop on the 0xC0000409 error, we need to attach a second instance of VS to the first instance of VS debugging the app. When you F5 the app on the first instance, it will load mscordbi.dll which contains the ShimProcess::HandleWin32DebugEvent function. I usually set a C++ function breakpoint then, when it hits it, I clear it and then set the conditional line breakpoint at the appropriate place.

kg · 2024-12-10T01:19:47Z

Main thread stack:

 # Child-SP          RetAddr               Call Site
00 00000089`a877d2e0 00007ffe`e287a2e1     ntdll!KiUserApcDispatch+0x50
01 00000089`a877dff0 00007ffe`e287bf33     0x00007ffe`e287a2e1
02 00000089`a877e030 00007ffe`e28818bf     0x00007ffe`e287bf33
03 00000089`a877e0d0 00007ffe`e28813eb     0x00007ffe`e28818bf
04 00000089`a877e2a0 00007ffe`e28810ec     0x00007ffe`e28813eb
05 00000089`a877e360 00007ffe`e2876acd     0x00007ffe`e28810ec
06 00000089`a877e430 00007fff`424369a3     0x00007ffe`e2876acd
07 00000089`a877ea10 00007fff`4239576c     coreclr!CallDescrWorkerInternal+0x83 [D:\a\_work\1\s\src\coreclr\vm\amd64\CallDescrWorkerAMD64.asm @ 74] 
08 00000089`a877ea50 00007fff`423b3d28     coreclr!MethodDescCallSite::CallTargetWorker+0x208 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp @ 595] 
09 (Inline Function) --------`--------     coreclr!MethodDescCallSite::Call+0xb [D:\a\_work\1\s\src\coreclr\vm\callhelpers.h @ 465] 
0a 00000089`a877eb90 00007fff`423b3e86     coreclr!RunMainInternal+0x11c [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp @ 1235] 
0b 00000089`a877ecb0 00007fff`423b419f     coreclr!RunMain+0xd2 [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp @ 1306] 
0c 00000089`a877ed60 00007fff`423a1077     coreclr!Assembly::ExecuteMainMethod+0x1a3 [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp @ 1434] 
0d 00000089`a877f030 00007fff`42360ecc     coreclr!CorHost2::ExecuteAssembly+0x1e7 [D:\a\_work\1\s\src\coreclr\vm\corhost.cpp @ 349] 
0e 00000089`a877f130 00007fff`54efec7f     coreclr!coreclr_execute_assembly+0xcc [D:\a\_work\1\s\src\coreclr\dlls\mscoree\exports.cpp @ 494] 
0f (Inline Function) --------`--------     hostpolicy!coreclr_t::execute_assembly+0x29 [D:\a\_work\1\s\src\native\corehost\hostpolicy\coreclr.cpp @ 108] 
10 00000089`a877f1d0 00007fff`54efef5c     hostpolicy!run_app_for_context+0x58f [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp @ 256] 
11 00000089`a877f2f0 00007fff`54eff86a     hostpolicy!run_app+0x3c [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp @ 285] 
12 00000089`a877f330 00007fff`5508da09     hostpolicy!corehost_main+0x15a [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp @ 426] 
13 00000089`a877f430 00007fff`5508ff86     hostfxr!execute_app+0x2e9 [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp @ 145] 
14 00000089`a877f510 00007fff`5509207c     hostfxr!`anonymous namespace'::read_config_and_execute+0xa6 [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp @ 532] 
15 00000089`a877f600 00007fff`55090553     hostfxr!fx_muxer_t::handle_exec_host_command+0x16c [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp @ 1007] 
16 00000089`a877f6b0 00007fff`55088390     hostfxr!fx_muxer_t::execute+0x483 [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp @ 578] 
17 00000089`a877f7f0 00007ff6`2fc2a4a1     hostfxr!hostfxr_main_startupinfo+0xa0 [D:\a\_work\1\s\src\native\corehost\fxr\hostfxr.cpp @ 63] 
18 00000089`a877f8f0 00007ff6`2fc2a8b6     Benchmarks_exe!exe_start+0x7b1 [D:\a\_work\1\s\src\native\corehost\corehost.cpp @ 253] 
19 00000089`a877faa0 00007ff6`2fc321f8     Benchmarks_exe!wmain+0x146 [D:\a\_work\1\s\src\native\corehost\corehost.cpp @ 324] 
1a (Inline Function) --------`--------     Benchmarks_exe!invoke_main+0x22 [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 90] 
1b 00000089`a877fb10 00007fff`bf51259d     Benchmarks_exe!__scrt_common_main_seh+0x10c [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288] 
1c 00000089`a877fb50 00007fff`c092af38     KERNEL32!BaseThreadInitThunk+0x1d
1d 00000089`a877fb80 00000000`00000000     ntdll!RtlUserThreadStart+0x28

I think maybe I need to disable tiered compilation to get a workable stack here... none of the other threads seem to be doing anything.

EDIT: Hmm, started it with TieredCompilation set to 0 and the managed frames are still bare symbols.

kg · 2024-12-10T01:22:01Z

Got it:

000000CD5997D0F0 00007ffec3044db3 coreclr!ClassLoader::LoadTypeDefThrowing + 0x103 [D:\a\_work\1\s\src\coreclr\vm\clsload.cpp:2014], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000CD5997D240 00007ffec312a0e1 coreclr!Frame::GetReturnAddress + 0x11 [D:\a\_work\1\s\src\coreclr\vm\frames.h:473], calling ntdll!LdrpDispatchUserCallTarget
000000CD5997DA60 00007ffe635cc1b3 (MethodDesc 00007ffe63892e70 + 0x183 SimdDictionary.VectorizedDictionary`2+LoopingBucketEnumerator[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].Retreat()), calling (MethodDesc 00007ffe63892988 + 0 System.Span`1[[SimdDictionary.VectorizedDictionary`2+Bucket[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]], SimdDictionary]].get_Item(Int32))
000000CD5997DA90 00007ffe635ca39f (MethodDesc 00007ffe63895a38 + 0x8f System.Collections.HashHelpers.FastMod(UInt32, UInt32, UInt64)), calling (MethodDesc 00007ffe63896500 + 0 System.Diagnostics.Debug.Assert(Boolean, System.String))
000000CD5997DB00 00007ffe635cb3be (MethodDesc 00007ffe637e61d8 + 0x42e SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].AdjustCascadeCounts(LoopingBucketEnumerator<System.__Canon,Int32>, Boolean)), calling 00007ffe637a9650
000000CD5997DB50 00007ffe635caf41 (MethodDesc 00007ffe6388fa50 + 0x61 SimdDictionary.VectorizedDictionary`2+Bucket[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].SetSlot(UIntPtr, Byte)), calling (MethodDesc 00007ffe63896500 + 0 System.Diagnostics.Debug.Assert(Boolean, System.String))
000000CD5997DC50 00007ffe635c9250 (MethodDesc 00007ffe638916e0 + 0x590 SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].TryInsert[[SimdDictionary.VectorizedDictionary`2+ComparerKeySearcher[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]], SimdDictionary]](System.__Canon, Int32, InsertMode<System.__Canon,Int32>, System.Collections.Generic.IEqualityComparer`1<System.__Canon>, InsertResult<System.__Canon,Int32> ByRef)), calling 00007ffe637a9038
000000CD5997DE80 00007ffe635c8c79 (MethodDesc 00007ffe637e6388 + 0x209 SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].TryInsert(System.__Canon, Int32, InsertMode<System.__Canon,Int32>, InsertResult<System.__Canon,Int32> ByRef)), calling 00007ffe637a8d68
000000CD5997DF50 00007ffe635c8983 (MethodDesc 00007ffe637e66e0 + 0x63 SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].TryAdd(System.__Canon, Int32)), calling 00007ffe637a8d50
000000CD5997DFD0 00007ffe635c87ee (MethodDesc 00007ffe637e66c8 + 0x6e SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].Add(System.__Canon, Int32)), calling 00007ffe637a8d20
000000CD5997E080 00007ffe635c69d9 (MethodDesc 00007ffe637e1a60 + 0x309 Benchmarks.Program.Main(System.String[])), calling 00007ffe637a2190
000000CD5997E660 00007ffec31669a3 coreclr!CallDescrWorkerInternal + 0x83 [D:\a\_work\1\s\src\coreclr\vm\amd64\CallDescrWorkerAMD64.asm:74]
000000CD5997E6A0 00007ffec30c576c coreclr!MethodDescCallSite::CallTargetWorker + 0x208 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:595], calling coreclr!CallDescrWorkerWithHandler [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:50]
000000CD5997E6E0 00007ffec30ec9b5 coreclr!IsAtProcessExit + 0x11 [D:\a\_work\1\s\src\coreclr\vm\vars.hpp:478], calling ntdll!LdrpDispatchUserCallTarget
000000CD5997E730 00007ffec30e3d5b coreclr!MethodDesc::IsVoid + 0x2f [D:\a\_work\1\s\src\coreclr\vm\method.cpp:1144], calling coreclr!SigPointer::PeekElemTypeClosed [D:\a\_work\1\s\src\coreclr\vm\siginfo.cpp:2426]
000000CD5997E7E0 00007ffec30e3d28 coreclr!RunMainInternal + 0x11c [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1235], calling coreclr!MethodDescCallSite::CallTargetWorker [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:287]
000000CD5997E800 00007ffec3103776 coreclr!MDInternalRO::GetSigOfMethodDef + 0x86 [D:\a\_work\1\s\src\coreclr\md\runtime\mdinternalro.cpp:1403], calling ntdll!LdrpDispatchUserCallTarget
000000CD5997E900 00007ffec30e3e86 coreclr!RunMain + 0xd2 [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1306], calling coreclr!RunMainInternal [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1206]
000000CD5997E9B0 00007ffec30e419f coreclr!Assembly::ExecuteMainMethod + 0x1a3 [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1434], calling coreclr!RunMain [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1255]
000000CD5997E9F0 00007ffec3044b2a coreclr!MetaSig::CompareMethodSigs + 0x1ba [D:\a\_work\1\s\src\coreclr\vm\siginfo.cpp:4352], calling coreclr!memcmp [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\string\amd64\memcmp.asm:64]
000000CD5997EA40 00007fffc090b001 ntdll!RtlFreeHeap + 0x51, calling ntdll!RtlpFreeHeapInternal
000000CD5997EAA0 00007ffec3167af5 coreclr!ThePreStub + 0x55 [D:\a\_work\1\s\src\coreclr\vm\amd64\ThePreStubAMD64.asm:21], calling coreclr!PreStubWorker [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp:2616]
000000CD5997EAB0 00007ffe8dba0c6a (MethodDesc 00007ffe63599f28 + 0x3a System.String.Ctor(Char*)), calling 00007ffe635317b8
000000CD5997EAF0 00007ffe8db8806f (MethodDesc 00007ffe636980b0 + 0xaf System.Environment.InitializeCommandLineArgs(Char*, Int32, Char**)), calling 00007ffe63470010
000000CD5997EB50 00007ffec31669a3 coreclr!CallDescrWorkerInternal + 0x83 [D:\a\_work\1\s\src\coreclr\vm\amd64\CallDescrWorkerAMD64.asm:74]
000000CD5997EB90 00007ffec30c54da coreclr!DispatchCallSimple + 0x72 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:249], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000CD5997EC30 00007ffec30934d0 coreclr!CLRConfig::GetConfigValue + 0x34 [D:\a\_work\1\s\src\coreclr\utilcode\clrconfig.cpp:451], calling coreclr!`anonymous namespace'::EnvGetString [D:\a\_work\1\s\src\coreclr\utilcode\clrconfig.cpp:134]
000000CD5997EC80 00007ffec30d1077 coreclr!CorHost2::ExecuteAssembly + 0x1e7 [D:\a\_work\1\s\src\coreclr\vm\corhost.cpp:349], calling coreclr!Assembly::ExecuteMainMethod [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1371]
000000CD5997ED80 00007ffec3090ecc coreclr!coreclr_execute_assembly + 0xcc [D:\a\_work\1\s\src\coreclr\dlls\mscoree\exports.cpp:494], calling ntdll!LdrpDispatchUserCallTarget
000000CD5997EDA0 00007fff54ee237a hostpolicy!breadcrumb_writer_t::begin_write + 0x1ba [D:\a\_work\1\s\src\native\corehost\hostpolicy\breadcrumbs.cpp:39], calling hostpolicy!trace::verbose [D:\a\_work\1\s\src\native\corehost\hostmisc\trace.cpp:131]
000000CD5997EDC0 00007fffbe238371 ucrtbase!is_stream_flushable_or_commitable + 0x9, calling ucrtbase!is_stream_flushable
000000CD5997EDF0 00007fffbe2389d7 ucrtbase!fflush + 0x17, calling ucrtbase!is_stream_flushable_or_commitable
000000CD5997EE20 00007fff54efec7f hostpolicy!run_app_for_context + 0x58f [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:256], calling ntdll!LdrpDispatchUserCallTarget
000000CD5997EF40 00007fff54efef5c hostpolicy!run_app + 0x3c [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:285], calling hostpolicy!run_app_for_context [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:208]
000000CD5997EF80 00007fff54eff86a hostpolicy!corehost_main + 0x15a [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:426], calling hostpolicy!run_app [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:280]
000000CD5997F080 00007fff5508da09 hostfxr!execute_app + 0x2e9 [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:145], calling ntdll!LdrpDispatchUserCallTarget
000000CD5997F160 00007fff5508ff86 hostfxr!`anonymous namespace'::read_config_and_execute + 0xa6 [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:532], calling hostfxr!execute_app [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:87]
000000CD5997F250 00007fff5509207c hostfxr!fx_muxer_t::handle_exec_host_command + 0x16c [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:1007], calling hostfxr!`anonymous namespace'::read_config_and_execute [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:515]
000000CD5997F300 00007fff55090553 hostfxr!fx_muxer_t::execute + 0x483 [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:578], calling hostfxr!fx_muxer_t::handle_exec_host_command [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:990]
000000CD5997F440 00007fff55088390 hostfxr!hostfxr_main_startupinfo + 0xa0 [D:\a\_work\1\s\src\native\corehost\fxr\hostfxr.cpp:63], calling hostfxr!fx_muxer_t::execute [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:555]
000000CD5997F460 00007ff62fc294fe Benchmarks_exe!hostfxr_resolver_t::hostfxr_resolver_t + 0x16e [D:\a\_work\1\s\src\native\corehost\apphost\standalone\hostfxr_resolver.cpp:132], calling Benchmarks_exe!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000CD5997F510 00007fffbe2389d7 ucrtbase!fflush + 0x17, calling ucrtbase!is_stream_flushable_or_commitable
000000CD5997F540 00007ff62fc2a4a1 Benchmarks_exe!exe_start + 0x7b1 [D:\a\_work\1\s\src\native\corehost\corehost.cpp:253], calling ntdll!LdrpDispatchUserCallTarget
000000CD5997F6F0 00007ff62fc2a8b6 Benchmarks_exe!wmain + 0x146 [D:\a\_work\1\s\src\native\corehost\corehost.cpp:324], calling Benchmarks_exe!exe_start [D:\a\_work\1\s\src\native\corehost\corehost.cpp:109]
000000CD5997F730 00007ff62fc31f5d Benchmarks_exe!__scrt_release_startup_lock + 0xd [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\utility\utility.cpp:161], calling Benchmarks_exe!__scrt_is_ucrt_dll_in_use [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\utility\ucrt_detection.c:22]
000000CD5997F760 00007ff62fc321f8 Benchmarks_exe!__scrt_common_main_seh + 0x10c [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288], calling Benchmarks_exe!wmain [D:\a\_work\1\s\src\native\corehost\corehost.cpp:303]
000000CD5997F770 00007ff62fc32279 Benchmarks_exe!wmainCRTStartup + 0x9 [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_wmain.cpp:17], calling Benchmarks_exe!__security_init_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\gs_support.c:158]
000000CD5997F7A0 00007fffbf51259d KERNEL32!BaseThreadInitThunk + 0x1d, calling KERNEL32!BaseThreadInitXfgThunk
000000CD5997F7D0 00007fffc092af38 ntdll!RtlUserThreadStart + 0x28, calling ntdll!LdrpDispatchUserCallTarget

Edit: Reproed again, another different !DumpStack trace:

000000E300F7D080 00007ffef45e4db3 coreclr!ClassLoader::LoadTypeDefThrowing + 0x103 [D:\a\_work\1\s\src\coreclr\vm\clsload.cpp:2014], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000E300F7D1A0 00007ffef46cc434 coreclr!TransitionFrame::GetReturnAddressPtr + 0x14 [D:\a\_work\1\s\src\coreclr\vm\frames.h:943], calling ntdll!LdrpDispatchUserCallTarget
000000E300F7D1B0 00007fffc0923810 ntdll!RtlSetLastWin32Error + 0x40, calling ntdll!_security_check_cookie
000000E300F7D1D0 00007ffef46ca0e1 coreclr!Frame::GetReturnAddress + 0x11 [D:\a\_work\1\s\src\coreclr\vm\frames.h:473], calling ntdll!LdrpDispatchUserCallTarget
000000E300F7D200 00007ffef45ce513 coreclr!ExternalMethodFixupWorker + 0x8f3 [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp:3407], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000E300F7DA90 00007ffe94b6b222 (MethodDesc 00007ffe94d86428 + 0x292 SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].AdjustCascadeCounts(LoopingBucketEnumerator<System.__Canon,Int32>, Boolean)), calling 00007ffe94d49668
000000E300F7DAE0 00007ffe94b6af41 (MethodDesc 00007ffe94e2fa50 + 0x61 SimdDictionary.VectorizedDictionary`2+Bucket[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].SetSlot(UIntPtr, Byte)), calling (MethodDesc 00007ffe94e36500 + 0 System.Diagnostics.Debug.Assert(Boolean, System.String))
000000E300F7DBE0 00007ffe94b69250 (MethodDesc 00007ffe94e316e0 + 0x590 SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].TryInsert[[SimdDictionary.VectorizedDictionary`2+ComparerKeySearcher[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]], SimdDictionary]](System.__Canon, Int32, InsertMode<System.__Canon,Int32>, System.Collections.Generic.IEqualityComparer`1<System.__Canon>, InsertResult<System.__Canon,Int32> ByRef)), calling 00007ffe94d49038
000000E300F7DE10 00007ffe94b68c79 (MethodDesc 00007ffe94d865d8 + 0x209 SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].TryInsert(System.__Canon, Int32, InsertMode<System.__Canon,Int32>, InsertResult<System.__Canon,Int32> ByRef)), calling 00007ffe94d48d68
000000E300F7DEE0 00007ffe94b68983 (MethodDesc 00007ffe94d86930 + 0x63 SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].TryAdd(System.__Canon, Int32)), calling 00007ffe94d48d50
000000E300F7DF60 00007ffe94b687ee (MethodDesc 00007ffe94d86918 + 0x6e SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].Add(System.__Canon, Int32)), calling 00007ffe94d48d20
000000E300F7E010 00007ffe94b669d9 (MethodDesc 00007ffe94d81cb0 + 0x309 Benchmarks.Program.Main(System.String[])), calling 00007ffe94d42190
000000E300F7E5F0 00007ffef47069a3 coreclr!CallDescrWorkerInternal + 0x83 [D:\a\_work\1\s\src\coreclr\vm\amd64\CallDescrWorkerAMD64.asm:74]
~snip~

tommcdon · 2024-12-10T02:24:09Z

@kg - thanks for the call stacks! The first one (from main), indicates that we were dispatching an APC. @thaystg recently produced a bug fix related to APC's here - #110533. It might be worth testing with the fix (I suggest a retail build as VS can trigger various asserts on debug builds). Note that if you try a private runtime build, please see https://aka.ms/vs/unsigned-dotnet-debugger-lib.

janvorli · 2024-12-11T13:17:58Z

@kg the 0xC0000409 is most likely a CET related issue. When the crash occurs, it would be helpful to dump the shadow stack too, it may reveal the culprit. You can use e.g. dps @ssp to do that.

kg · 2024-12-11T17:09:25Z

From the minidump I took:

For analysis of this file, run !analyze -v
ntdll!KiUserApcDispatch+0x50:
00007fff`c0974120 cd29            int     29h
0:000> dps @ssp
000000e3`010fef30  00007ffe`94b6b229
000000e3`010fef38  00007ffe`94b69250
000000e3`010fef40  00007ffe`94b68c79
000000e3`010fef48  00007ffe`94b68983
000000e3`010fef50  00007ffe`94b687ee
000000e3`010fef58  00007ffe`94b669d9
000000e3`010fef60  00007ffe`f47069a3 coreclr!CallDescrWorkerInternal+0x83 [D:\a\_work\1\s\src\coreclr\vm\amd64\CallDescrWorkerAMD64.asm @ 74]
000000e3`010fef68  00007ffe`f466576c coreclr!MethodDescCallSite::CallTargetWorker+0x208 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp @ 595]
000000e3`010fef70  00007ffe`f4683d28 coreclr!RunMainInternal+0x11c [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp @ 1235]
000000e3`010fef78  00007ffe`f4683e86 coreclr!RunMain+0xd2 [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp @ 1306]
000000e3`010fef80  00007ffe`f468419f coreclr!Assembly::ExecuteMainMethod+0x1a3 [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp @ 1434]
000000e3`010fef88  00007ffe`f4671077 coreclr!CorHost2::ExecuteAssembly+0x1e7 [D:\a\_work\1\s\src\coreclr\vm\corhost.cpp @ 349]
000000e3`010fef90  00007ffe`f4630ecc coreclr!coreclr_execute_assembly+0xcc [D:\a\_work\1\s\src\coreclr\dlls\mscoree\exports.cpp @ 494]
000000e3`010fef98  00007fff`54efec7f hostpolicy!run_app_for_context+0x58f [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp @ 256]
000000e3`010fefa0  00007fff`54efef5c hostpolicy!run_app+0x3c [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp @ 285]
000000e3`010fefa8  00007fff`54eff86a hostpolicy!corehost_main+0x15a [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp @ 426]

janvorli · 2024-12-11T20:21:54Z

@kg can you please dump the regular stack trace too? This doesn't seem to match the return addresses on the regular stack you've dumped in the comment #110552 (comment) above, so I assume it is a different run.

janvorli · 2024-12-11T20:23:42Z

It would be also better to use !clrstack -f to dump the stack trace so that we can see the managed methods too.

kg · 2024-12-11T20:28:12Z

They match the 'another dumpstack trace' from the previous comment and are from a minidump I captured. Maybe minidumps don't work right? !clrstack -f doesn't seem to work.

kg · 2024-12-11T20:34:53Z

Attempting to repro again:
Exception thrown at 0x00007FFEB0FB7F28 (Microsoft.CodeAnalysis.ExpressionEvaluator.ResultProvider.ni.dll) in devenv.exe: 0xC0000005: Access violation reading location 0x0000000000000000.

On the second try I got a repro. dps @ssp:

000000e6`11cfef30  00007ffe`48212ef0
000000e6`11cfef38  00007ffe`48210ef0
000000e6`11cfef40  00007ffe`482108cb
000000e6`11cfef48  00007ffe`482105b3
000000e6`11cfef50  00007ffe`4821041e
000000e6`11cfef58  00007ffe`4820e5a9
000000e6`11cfef60  00007ffe`a7da69a3 coreclr!CallDescrWorkerInternal+0x83 [D:\a\_work\1\s\src\coreclr\vm\amd64\CallDescrWorkerAMD64.asm @ 74]
000000e6`11cfef68  00007ffe`a7d0576c coreclr!MethodDescCallSite::CallTargetWorker+0x208 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp @ 595]
000000e6`11cfef70  00007ffe`a7d23d28 coreclr!RunMainInternal+0x11c [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp @ 1235]
000000e6`11cfef78  00007ffe`a7d23e86 coreclr!RunMain+0xd2 [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp @ 1306]
000000e6`11cfef80  00007ffe`a7d2419f coreclr!Assembly::ExecuteMainMethod+0x1a3 [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp @ 1434]
000000e6`11cfef88  00007ffe`a7d11077 coreclr!CorHost2::ExecuteAssembly+0x1e7 [D:\a\_work\1\s\src\coreclr\vm\corhost.cpp @ 349]
000000e6`11cfef90  00007ffe`a7cd0ecc coreclr!coreclr_execute_assembly+0xcc [D:\a\_work\1\s\src\coreclr\dlls\mscoree\exports.cpp @ 494]
000000e6`11cfef98  00007ffe`b01eec7f hostpolicy!run_app_for_context+0x58f [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp @ 256]
000000e6`11cfefa0  00007ffe`b01eef5c hostpolicy!run_app+0x3c [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp @ 285]
000000e6`11cfefa8  00007ffe`b01ef86a hostpolicy!corehost_main+0x15a [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp @ 426]

!clrstack -f

0:000> !clrstack -f
OS Thread Id: 0xbe28 (0)
        Child SP               IP Call Site
000000E611B7CEA0 00007FFFC0974120 ntdll!KiUserApcDispatch + 80
000000E611B7DBA0 00007FFE48212EF0 
000000E611B7DCF0 00007FFE48210EF0 
000000E611B7DF20 00007FFE482108CB 
000000E611B7E000 00007FFE482105B3 
000000E611B7E080 00007FFE4821041E 
000000E611B7E130 00007FFE4820E5A9 
000000E611B7E710 00007FFEA7DA69A3 coreclr!CallDescrWorkerInternal + 131 at D:\a\_work\1\s\src\coreclr\vm\amd64\CallDescrWorkerAMD64.asm:74
000000E611B7E750 00007FFEA7D0576C coreclr!MethodDescCallSite::CallTargetWorker + 520 at D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:595
000000E611B7E890 00007FFEA7D23D28 coreclr!RunMainInternal + 284 at D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1235
000000E611B7E9B0 00007FFEA7D23E86 coreclr!RunMain + 210 at D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1306
000000E611B7EA60 00007FFEA7D2419F coreclr!Assembly::ExecuteMainMethod + 419 at D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1434
000000E611B7ED30 00007FFEA7D11077 coreclr!CorHost2::ExecuteAssembly + 487 at D:\a\_work\1\s\src\coreclr\vm\corhost.cpp:349
000000E611B7EE30 00007FFEA7CD0ECC coreclr!coreclr_execute_assembly + 204 at D:\a\_work\1\s\src\coreclr\dlls\mscoree\exports.cpp:494
000000E611B7EED0 00007FFEB01EEC7F hostpolicy!run_app_for_context + 1423 at D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:256
000000E611B7EFF0 00007FFEB01EEF5C hostpolicy!run_app + 60 at D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:285
000000E611B7F030 00007FFEB01EF86A hostpolicy!corehost_main + 346 at D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:426
000000E611B7F130 00007FFEB023DA09 hostfxr!execute_app + 745 at D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:145
000000E611B7F210 00007FFEB023FF86 hostfxr!`anonymous namespace'::read_config_and_execute + 166 at D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:532
000000E611B7F300 00007FFEB024207C hostfxr!fx_muxer_t::handle_exec_host_command + 364 at D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:1007
000000E611B7F3B0 00007FFEB0240553 hostfxr!fx_muxer_t::execute + 1155 at D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:578
000000E611B7F4F0 00007FFEB0238390 hostfxr!hostfxr_main_startupinfo + 160 at D:\a\_work\1\s\src\native\corehost\fxr\hostfxr.cpp:63
000000E611B7F5F0 00007FF6849CA4A1 Benchmarks_exe!exe_start + 1969 at D:\a\_work\1\s\src\native\corehost\corehost.cpp:253
000000E611B7F7A0 00007FF6849CA8B6 Benchmarks_exe!wmain + 326 at D:\a\_work\1\s\src\native\corehost\corehost.cpp:324
000000E611B7F810 00007FF6849D21F8 Benchmarks_exe!__scrt_common_main_seh + 268 at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
000000E611B7F850 00007FFFBF51259D KERNEL32!BaseThreadInitThunk + 29
000000E611B7F880 00007FFFC092AF38 ntdll!RtlUserThreadStart + 40

!DumpStack

0:000> !DumpStack
OS Thread Id: 0xbe28 (0)
Current frame: ntdll!KiUserApcDispatch + 0x50
Child-SP         RetAddr          Caller, Callee
000000E611B7D180 00007ffea7c84db3 coreclr!ClassLoader::LoadTypeDefThrowing + 0x103 [D:\a\_work\1\s\src\coreclr\vm\clsload.cpp:2014], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000E611B7DB90 00007ffe48212ebd (MethodDesc 00007ffe484d5578 + 0x1dd SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].AdjustCascadeCounts(LoopingBucketEnumerator<System.__Canon,Int32>, Boolean)), calling 00007ffe484c8b40
000000E611B7DBE0 00007ffe48212c91 (MethodDesc 00007ffe48563f88 + 0x61 SimdDictionary.VectorizedDictionary`2+Bucket[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].SetSlot(UIntPtr, Byte)), calling (MethodDesc 00007ffe4856aa38 + 0 System.Diagnostics.Debug.Assert(Boolean, System.String))
000000E611B7DCE0 00007ffe48210ef0 (MethodDesc 00007ffe48565ba0 + 0x590 SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].TryInsert[[SimdDictionary.VectorizedDictionary`2+ComparerKeySearcher[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]], SimdDictionary]](System.__Canon, Int32, InsertMode<System.__Canon,Int32>, System.Collections.Generic.IEqualityComparer`1<System.__Canon>, InsertResult<System.__Canon,Int32> ByRef)), calling 00007ffe484c8510
000000E611B7DF10 00007ffe482108cb (MethodDesc 00007ffe484d5770 + 0x22b SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].TryInsert(System.__Canon, Int32, InsertMode<System.__Canon,Int32>, InsertResult<System.__Canon,Int32> ByRef)), calling 00007ffe484c8228
000000E611B7DFF0 00007ffe482105b3 (MethodDesc 00007ffe484d5ac8 + 0x63 SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].TryAdd(System.__Canon, Int32)), calling 00007ffe484c8210
000000E611B7E070 00007ffe4821041e (MethodDesc 00007ffe484d5ab0 + 0x6e SimdDictionary.VectorizedDictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].Add(System.__Canon, Int32)), calling 00007ffe484c81e0
000000E611B7E120 00007ffe4820e5a9 (MethodDesc 00007ffe484218f0 + 0x309 Benchmarks.Program.Main(System.String[])), calling 00007ffe484c1a88
000000E611B7E700 00007ffea7da69a3 coreclr!CallDescrWorkerInternal + 0x83 [D:\a\_work\1\s\src\coreclr\vm\amd64\CallDescrWorkerAMD64.asm:74]
000000E611B7E740 00007ffea7d0576c coreclr!MethodDescCallSite::CallTargetWorker + 0x208 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:595], calling coreclr!CallDescrWorkerWithHandler [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:50]
000000E611B7E780 00007ffea7d2c9b5 coreclr!IsAtProcessExit + 0x11 [D:\a\_work\1\s\src\coreclr\vm\vars.hpp:478], calling ntdll!LdrpDispatchUserCallTarget
000000E611B7E7D0 00007ffea7d23d5b coreclr!MethodDesc::IsVoid + 0x2f [D:\a\_work\1\s\src\coreclr\vm\method.cpp:1144], calling coreclr!SigPointer::PeekElemTypeClosed [D:\a\_work\1\s\src\coreclr\vm\siginfo.cpp:2426]
000000E611B7E880 00007ffea7d23d28 coreclr!RunMainInternal + 0x11c [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1235], calling coreclr!MethodDescCallSite::CallTargetWorker [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:287]
000000E611B7E8A0 00007ffea7d43776 coreclr!MDInternalRO::GetSigOfMethodDef + 0x86 [D:\a\_work\1\s\src\coreclr\md\runtime\mdinternalro.cpp:1403], calling ntdll!LdrpDispatchUserCallTarget
000000E611B7E9A0 00007ffea7d23e86 coreclr!RunMain + 0xd2 [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1306], calling coreclr!RunMainInternal [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1206]
000000E611B7EA50 00007ffea7d2419f coreclr!Assembly::ExecuteMainMethod + 0x1a3 [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1434], calling coreclr!RunMain [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1255]
000000E611B7EA90 00007ffea7c84b2a coreclr!MetaSig::CompareMethodSigs + 0x1ba [D:\a\_work\1\s\src\coreclr\vm\siginfo.cpp:4352], calling coreclr!memcmp [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\string\amd64\memcmp.asm:64]
000000E611B7EAE0 00007fffc090b001 ntdll!RtlFreeHeap + 0x51, calling ntdll!RtlpFreeHeapInternal
000000E611B7EB40 00007ffea7da7af5 coreclr!ThePreStub + 0x55 [D:\a\_work\1\s\src\coreclr\vm\amd64\ThePreStubAMD64.asm:21], calling coreclr!PreStubWorker [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp:2616]
000000E611B7EB50 00007ffea0790c6a (MethodDesc 00007ffe481d9f28 + 0x3a System.String.Ctor(Char*)), calling 00007ffe481717b8
000000E611B7EB90 00007ffea077806f (MethodDesc 00007ffe482d80b0 + 0xaf System.Environment.InitializeCommandLineArgs(Char*, Int32, Char**)), calling 00007ffe480b0010
000000E611B7EBF0 00007ffea7da69a3 coreclr!CallDescrWorkerInternal + 0x83 [D:\a\_work\1\s\src\coreclr\vm\amd64\CallDescrWorkerAMD64.asm:74]
000000E611B7EC30 00007ffea7d054da coreclr!DispatchCallSimple + 0x72 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:249], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000E611B7ECD0 00007ffea7cd34d0 coreclr!CLRConfig::GetConfigValue + 0x34 [D:\a\_work\1\s\src\coreclr\utilcode\clrconfig.cpp:451], calling coreclr!`anonymous namespace'::EnvGetString [D:\a\_work\1\s\src\coreclr\utilcode\clrconfig.cpp:134]
000000E611B7ED20 00007ffea7d11077 coreclr!CorHost2::ExecuteAssembly + 0x1e7 [D:\a\_work\1\s\src\coreclr\vm\corhost.cpp:349], calling coreclr!Assembly::ExecuteMainMethod [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1371]
000000E611B7EE20 00007ffea7cd0ecc coreclr!coreclr_execute_assembly + 0xcc [D:\a\_work\1\s\src\coreclr\dlls\mscoree\exports.cpp:494], calling ntdll!LdrpDispatchUserCallTarget
000000E611B7EE40 00007ffeb01d237a hostpolicy!breadcrumb_writer_t::begin_write + 0x1ba [D:\a\_work\1\s\src\native\corehost\hostpolicy\breadcrumbs.cpp:39], calling hostpolicy!trace::verbose [D:\a\_work\1\s\src\native\corehost\hostmisc\trace.cpp:131]
000000E611B7EE60 00007fffbe238371 ucrtbase!is_stream_flushable_or_commitable + 0x9, calling ucrtbase!is_stream_flushable
000000E611B7EE90 00007fffbe2389d7 ucrtbase!fflush + 0x17, calling ucrtbase!is_stream_flushable_or_commitable
000000E611B7EEC0 00007ffeb01eec7f hostpolicy!run_app_for_context + 0x58f [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:256], calling ntdll!LdrpDispatchUserCallTarget
000000E611B7EFE0 00007ffeb01eef5c hostpolicy!run_app + 0x3c [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:285], calling hostpolicy!run_app_for_context [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:208]
000000E611B7F020 00007ffeb01ef86a hostpolicy!corehost_main + 0x15a [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:426], calling hostpolicy!run_app [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:280]
000000E611B7F120 00007ffeb023da09 hostfxr!execute_app + 0x2e9 [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:145], calling ntdll!LdrpDispatchUserCallTarget
000000E611B7F200 00007ffeb023ff86 hostfxr!`anonymous namespace'::read_config_and_execute + 0xa6 [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:532], calling hostfxr!execute_app [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:87]
000000E611B7F2F0 00007ffeb024207c hostfxr!fx_muxer_t::handle_exec_host_command + 0x16c [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:1007], calling hostfxr!`anonymous namespace'::read_config_and_execute [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:515]
000000E611B7F3A0 00007ffeb0240553 hostfxr!fx_muxer_t::execute + 0x483 [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:578], calling hostfxr!fx_muxer_t::handle_exec_host_command [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:990]
000000E611B7F4E0 00007ffeb0238390 hostfxr!hostfxr_main_startupinfo + 0xa0 [D:\a\_work\1\s\src\native\corehost\fxr\hostfxr.cpp:63], calling hostfxr!fx_muxer_t::execute [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:555]
000000E611B7F500 00007ff6849c94fe Benchmarks_exe!hostfxr_resolver_t::hostfxr_resolver_t + 0x16e [D:\a\_work\1\s\src\native\corehost\apphost\standalone\hostfxr_resolver.cpp:132], calling Benchmarks_exe!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000E611B7F5B0 00007fffbe2389d7 ucrtbase!fflush + 0x17, calling ucrtbase!is_stream_flushable_or_commitable
000000E611B7F5E0 00007ff6849ca4a1 Benchmarks_exe!exe_start + 0x7b1 [D:\a\_work\1\s\src\native\corehost\corehost.cpp:253], calling ntdll!LdrpDispatchUserCallTarget
000000E611B7F790 00007ff6849ca8b6 Benchmarks_exe!wmain + 0x146 [D:\a\_work\1\s\src\native\corehost\corehost.cpp:324], calling Benchmarks_exe!exe_start [D:\a\_work\1\s\src\native\corehost\corehost.cpp:109]
000000E611B7F7D0 00007ff6849d1f5d Benchmarks_exe!__scrt_release_startup_lock + 0xd [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\utility\utility.cpp:161], calling Benchmarks_exe!__scrt_is_ucrt_dll_in_use [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\utility\ucrt_detection.c:22]
000000E611B7F800 00007ff6849d21f8 Benchmarks_exe!__scrt_common_main_seh + 0x10c [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288], calling Benchmarks_exe!wmain [D:\a\_work\1\s\src\native\corehost\corehost.cpp:303]
000000E611B7F810 00007ff6849d2279 Benchmarks_exe!wmainCRTStartup + 0x9 [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_wmain.cpp:17], calling Benchmarks_exe!__security_init_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\gs_support.c:158]
000000E611B7F840 00007fffbf51259d KERNEL32!BaseThreadInitThunk + 0x1d, calling KERNEL32!BaseThreadInitXfgThunk
000000E611B7F870 00007fffc092af38 ntdll!RtlUserThreadStart + 0x28, calling ntdll!LdrpDispatchUserCallTarget

I have a minidump of this instance if you want it.

janvorli · 2024-12-11T21:00:01Z

@kg hmm, the shadow stack seems to match the real one, so that's strange. Please share the dump, I'll take a look to see if I can spot something.

kg · 2024-12-11T21:03:25Z

Wouldn't upload as a github attachment so here's a dropbox link.
https://www.dropbox.com/scl/fi/6n1hvmla5j9ep1vevc7e7/debugee.zip?rlkey=hzeawe6xtbyvs0lyte9iq0een&dl=0

janvorli · 2024-12-11T21:40:35Z

I can see what's wrong. The special user mode APC that we use for runtime suspension was just about to return (it restores a context to do that), but the Rip in the context that it wants to restore to doesn't match the frame below the KiUserApcDispatch, but it is the address of FuncEvalHijack. So, it seems there is a race / bug between the funceval hijacking and the special user mode APC.
The special user mode APC works as follows. When we want to suspend a runtime, we send the user mode APC to all threads using the QueueUserAPC2 Windows API. Windows kernel then redirects the target thread to KiUserApcDispatch that calls our ApcActivationCallbackStub which further calls Thread::ApcActivationCallback. The callback in the case that happened in your run ends up waiting until the runtime is resumed again and then it returns back to the managed code.

The funceval hijack is performed in Debugger::FuncEvalSetup. It occurs when the runtime is suspended, and it patches the Rip in the context of the suspended thread to the FuncEvalHijack. I am not 100% sure on how it works further, but it seems that in our case, we resume the runtime with the FuncEvalHijack in the context. When CET is enabled, Windows don't allow setting a context to a location that's not on the shadow stack (or in a list of registered allowed continuation locations in the executable - used for EH or it was not registered by a call to SetProcessDynamicEHContinuationTargets, which can only be called out of process).

Maybe our debugger would need to register the FuncEvalHijack via the SetProcessDynamicEHContinuationTargets to fix the problem, but I'll let diagnostics folks to reason about this.

cc: @tommcdon

ato321 · 2025-01-06T10:51:38Z

I am not sure if this is same problem, but I have noticed under .NET9 step-by-step debugging fails if any other thread throws a exception (user handled). I made a small test program and if you put the break line after the thread starting, then you would notice that sub-sequent steps would not be breaking anymore after exception thrown in the running thread.
However, changing this project to .NET8 makes debugging work just fine.

namespace ConsoleApp42
{
    internal class Program
    {
        static void GenerateExceptions()
        {
            while (true)
            {
                try
                {
                    Thread.Sleep(1);
                    throw new Exception("This is an exception");
                }
                catch (Exception e)
                {
                    Console.WriteLine(e.Message);
                }
            }
        }

        static void Main(string[] args)
        {
            Console.WriteLine("Hello, World!");
            (new Thread( GenerateExceptions)).Start();
            Console.WriteLine("Starting meaningless function...");

            var list = new List<int>();
            for (var i = 0; i < 100; i++)
            {
                list.Add(i * 2);
            }

            var uselessValue = "Not needed";
            var random = new Random();

            foreach (var item in list)
            {
                var temp = item + random.Next(0, 100);
                if (temp % 2 == 0)
                {
                    Console.WriteLine("Even value: " + temp);
                }
                else
                {
                    Console.WriteLine("Odd value: " + temp);
                }
            }

            for (var i = 0; i < 10; i++)
            {
                var notUsed = i * 5;
            }

            var moreRandom = new List<int>();
            for (var i = 0; i < 50; i++)
            {
                moreRandom.Add(random.Next(0, 100));
            }

            foreach (var number in moreRandom)
            {
                if (number > 25)
                {
                    var meaninglessString = "This value is high: " + number;
                    Console.WriteLine(meaninglessString);
                }
            }

            var redundantList = new List<string>();
            for (var i = 0; i < 20; i++)
            {
                redundantList.Add("Entry: " + i);
            }

            foreach (var str in redundantList)
            {
                if (str.Contains("1"))
                {
                    Console.WriteLine("Found 1 in: " + str);
                }
            }
        }
    }
}

dotnet-issue-labeler bot added the area-Diagnostics-coreclr label Dec 9, 2024

dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Dec 9, 2024

tommcdon added this to the 10.0.0 milestone Dec 12, 2024

tommcdon assigned thaystg Dec 12, 2024

tommcdon removed the untriaged New issue has not been triaged by the area owner label Dec 12, 2024

thaystg linked a pull request Jan 14, 2025 that will close this issue

[debugger] Fix crash during Async Break when APC and CET are enabled #111408

Open

dotnet-policy-service bot added the in-pr There is an active PR which will close this issue when it is merged label Jan 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crashes during debugging when pausing in VS2022/.NET9 #110552

Crashes during debugging when pausing in VS2022/.NET9 #110552

kg commented Dec 9, 2024 •

edited

Loading

dotnet-policy-service bot commented Dec 9, 2024

kg commented Dec 9, 2024 •

edited

Loading

tommcdon commented Dec 10, 2024

kg commented Dec 10, 2024 •

edited

Loading

tommcdon commented Dec 10, 2024

kg commented Dec 10, 2024 •

edited

Loading

kg commented Dec 10, 2024 •

edited

Loading

tommcdon commented Dec 10, 2024 •

edited

Loading

janvorli commented Dec 11, 2024

kg commented Dec 11, 2024

janvorli commented Dec 11, 2024

janvorli commented Dec 11, 2024

kg commented Dec 11, 2024

kg commented Dec 11, 2024

janvorli commented Dec 11, 2024

kg commented Dec 11, 2024

janvorli commented Dec 11, 2024

ato321 commented Jan 6, 2025 •

edited

Loading

Crashes during debugging when pausing in VS2022/.NET9 #110552

Crashes during debugging when pausing in VS2022/.NET9 #110552

Comments

kg commented Dec 9, 2024 • edited Loading

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

dotnet-policy-service bot commented Dec 9, 2024

kg commented Dec 9, 2024 • edited Loading

tommcdon commented Dec 10, 2024

kg commented Dec 10, 2024 • edited Loading

tommcdon commented Dec 10, 2024

kg commented Dec 10, 2024 • edited Loading

kg commented Dec 10, 2024 • edited Loading

tommcdon commented Dec 10, 2024 • edited Loading

janvorli commented Dec 11, 2024

kg commented Dec 11, 2024

janvorli commented Dec 11, 2024

janvorli commented Dec 11, 2024

kg commented Dec 11, 2024

kg commented Dec 11, 2024

janvorli commented Dec 11, 2024

kg commented Dec 11, 2024

janvorli commented Dec 11, 2024

ato321 commented Jan 6, 2025 • edited Loading

kg commented Dec 9, 2024 •

edited

Loading

kg commented Dec 9, 2024 •

edited

Loading

kg commented Dec 10, 2024 •

edited

Loading

kg commented Dec 10, 2024 •

edited

Loading

kg commented Dec 10, 2024 •

edited

Loading

tommcdon commented Dec 10, 2024 •

edited

Loading

ato321 commented Jan 6, 2025 •

edited

Loading