Debugging Crossgen2 documentation #47363
Conversation
- Also new --print-repro-instructions command line switch to crossgen2 as requested by the JIT team
|
@dotnet/crossgen-contrib @dotnet/jit-contrib |
| - When debugging a multi-threaded component of Crossgen2 and not investigating a multi-threading issue itself, it is generally advisable to disable the use of multiple threads. | ||
| To do this use the `--parallelism 1` switch to specify that the maximum parallelism of the process shall be 1. | ||
|
|
||
| - When debugging the behaviour of compiling a single method, the compiler may be instructed to only compile a single method. This is done via the various --singlemethod options |
There was a problem hiding this comment.
Nit - exploring British spelling (behaviour) :D?
There was a problem hiding this comment.
:) More like I recognise both forms and so I tend to gloss over such details. Also, my spell checker didn't notice. :)
|
|
||
| - Passing special jit behavior flags to the compiler is done via the `--codegenopt` switch. As an example to turn on tailcall loop optimizations and dump all code compiled use a pair of them like `--codegenopt NgenDump=* --codegenopt TailCallLoopOpt=1`. | ||
|
|
||
| - When using the NgenDump feature of the JIT, disable parallelism as described above or specify a single method to be compiled. Otherwise, output from multiple functions will be interleaved and inscrutable. |
There was a problem hiding this comment.
The JIT has a COMPlus_JitStdOutFile argument. Maybe the JIT should allow something like "{threadid}" in the name and crossgen2 tell us some unique compilation ID that we replace that with, to generate a unique filename.
There was a problem hiding this comment.
Crossgen2 has no reliable way to produce a unique compilation id for a given method. Unlike crossgen.exe method load order, and other details are sufficiently non-deterministic as to make that impossible without disabling parallelization. However, details like threadid are visible to the jit as it is just a normal bit of native code. The difficulty in reading the output comes from the mess of printf messages printed by the JIT on all the various threads. If you modify the JIT to have some sort of output that makes sense when the jit is run in parallel, please update this document with the technique.
There was a problem hiding this comment.
Crossgen2 has no reliable way to produce a unique compilation id for a given method.
Interesting...
In this imagined scenario, it wouldn't need to be an id unique to the function (such that the same function always gets the same id).
Presumably the JIT could use some static global counter to incorporate into the name, as long as the JIT process itself doesn't get unloaded/re-loaded during compilation.
|
|
||
| - When using the NgenDump feature of the JIT, disable parallelism as described above or specify a single method to be compiled. Otherwise, output from multiple functions will be interleaved and inscrutable. | ||
|
|
||
| - Since there are 2 jits in the process, when debugging in the JIT, if the souce files match up, there is a decent chance that a native debugger will stop at unfortunate and unexpected locations. This is extremely annoying, and to combat this, we generally recommend making a point of using a runtime which doesn't exactly match that of the crossgen2 in use. However, if that isn't feasible, it is also possible to disable symbol loading in most native debuggers. For instance, in Visual Studio, one would use the "Specify excluded modules" feature. |
|
|
||
| - When using the NgenDump feature of the JIT, disable parallelism as described above or specify a single method to be compiled. Otherwise, output from multiple functions will be interleaved and inscrutable. | ||
|
|
||
| - Since there are 2 jits in the process, when debugging in the JIT, if the souce files match up, there is a decent chance that a native debugger will stop at unfortunate and unexpected locations. This is extremely annoying, and to combat this, we generally recommend making a point of using a runtime which doesn't exactly match that of the crossgen2 in use. However, if that isn't feasible, it is also possible to disable symbol loading in most native debuggers. For instance, in Visual Studio, one would use the "Specify excluded modules" feature. |
There was a problem hiding this comment.
How does crossgen2 choose and find the JIT? How can you override which JIT is used?
There was a problem hiding this comment.
Crossgen2 will find the jit to use by looking in the same directory as the crossgen2.dll binary. It uses a naming convention to identify the exact jit dll to use.
In addition, there is support for a --jitpath switch to use a specific JIT. This option is intended to support A/B testing by the JIT team. The --jitpath option should only be used if the jit interface has not been changed. The JIT specified by the --jitpath switch must be compatible with the current settings of the --targetos and --targetarch switches.
|
|
||
| - When using the NgenDump feature of the JIT, disable parallelism as described above or specify a single method to be compiled. Otherwise, output from multiple functions will be interleaved and inscrutable. | ||
|
|
||
| - Since there are 2 jits in the process, when debugging in the JIT, if the souce files match up, there is a decent chance that a native debugger will stop at unfortunate and unexpected locations. This is extremely annoying, and to combat this, we generally recommend making a point of using a runtime which doesn't exactly match that of the crossgen2 in use. However, if that isn't feasible, it is also possible to disable symbol loading in most native debuggers. For instance, in Visual Studio, one would use the "Specify excluded modules" feature. |
There was a problem hiding this comment.
Can crossgen2 itself be precompiled so you don't get the "JIT that compiled crossgen2" problem (most of the time)?
There was a problem hiding this comment.
Not really. The NativeAot work will be able to compile crossgen2, but that isn't ready yet. For production use we will compile it with itself, but our internal dev/test environment doesn't do that. Also, when R2R'd crossgen2 still has a fair amount of methods that need jitting.
|
|
||
| - Since there are 2 jits in the process, when debugging in the JIT, if the souce files match up, there is a decent chance that a native debugger will stop at unfortunate and unexpected locations. This is extremely annoying, and to combat this, we generally recommend making a point of using a runtime which doesn't exactly match that of the crossgen2 in use. However, if that isn't feasible, it is also possible to disable symbol loading in most native debuggers. For instance, in Visual Studio, one would use the "Specify excluded modules" feature. | ||
|
|
||
| - In parallel to the crossgen2 project, there has been an effort to build a tool known as r2rdump. This tool can be used to dump the contents of a produced image to examine what was actually produced in the final binary. It has a large multitude of options to control exactly what is dumped, but in general it is able to dump any image produced by crossgen2, and display its contents in a human readable fashion. Specify `--disasm` to display disassembly. |
There was a problem hiding this comment.
suggestion: has been an effort to build => is
|
|
||
| - Diagnosing why a specific method failed to compile in crossgen2 can be done by passing the `--verbose` switch to crossgen2. This will print many things, but in particular it will print the reason why a compilation was aborted due to an R2R format limitation. | ||
|
|
||
| - In the runtime testbed, each test can be commanded to compile with crossgen2 by using environment variables. Just set the `RunCrossgen2` variable to 1, and optionally set the `CompositeBuildMode` variable to 1 if you wish to see the R2R behavior with composite image creation. This is often the easiest way to run a simple test with crossgen2 for developers practiced in the CoreCLR testbed. |
There was a problem hiding this comment.
Of course, this requires invoking the test with the batch script wrapper.
There was a problem hiding this comment.
Ah, I skipped a step here. Once the script has been run using the batch script wrapper, a response file is produced which makes it easy to launch crossgen2 manually, and to describe the behavior of the __TestDotNetCmd environment variable, which is commonly needed to successfully run crossgen2 using the test batch script. I'll write up some explanatory text here, but here is the example I'm going to drop into the doc.
The batch script will print out the command line used to invoke crossgen2. For instance
C:\git2\runtime>set RunCrossgen2=1
C:\git2\runtime>c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\Complex1.cmd
BEGIN EXECUTION
Complex1.dll
1 file(s) copied.
Could Not Find c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\Complex1.dll.rsp
Response file: c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\\Complex1.dll.rsp
c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\IL-CG2\Complex1.dll
-o:c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\\Complex1.dll
--targetarch:x64
--verify-type-and-field-layout
-O
-r:c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\Tests\Core_Root\System.*.dll
-r:c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\Tests\Core_Root\Microsoft.*.dll
-r:c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\Tests\Core_Root\mscorlib.dll
-r:c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\Tests\Core_Root\netstandard.dll
" "dotnet" "c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\Tests\Core_Root\crossgen2\crossgen2.dll" @"c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\\Complex1.dll.rsp" -r:c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\IL-CG2\*.dll"
Emitting R2R PE file: c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\\Complex1.dll
"c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\Tests\Core_Root\corerun.exe" Complex1.dll
Starting...
Everything Worked!
Expected: 100
Actual: 100
END EXECUTION - PASSED
PASSED
From that invocation you can see that crossgen2 was launched with a response file containing a list of arguments including all of the details for references. Then you can manually run the actual crossgen2 command, which is prefixed with the value of the __TestDotNetCmd environment variable. For instance, once I saw the above output, I copied and pasted the last command, and ran it.
C:\git2\runtime>"dotnet" "c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\Tests\Core_Root\crossgen2\crossgen2.dll" @"c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\\Complex1.dll.rsp" -r:c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\IL-CG2\*.dll
C:\git2\runtime\.dotnet
Emitting R2R PE file: c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\\Complex1.dll
And then wanted to debug the individual method compilation, and ran it with the --print-repro-instructions switch
C:\git2\runtime>"dotnet" "c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\Tests\Core_Root\crossgen2\crossgen2.dll" @"c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\\Complex1.dll.rsp" -r:c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\IL-CG2\*.dll --print-repro-instructions
C:\git2\runtime\.dotnet
Single method repro args:--singlemethodtypename "Complex,Complex1" --singlemethodname mul_em --singlemethodindex 1
Single method repro args:--singlemethodtypename "Complex_Array_Test,Complex1" --singlemethodname .ctor --singlemethodindex 1
Single method repro args:--singlemethodtypename "Complex_Array_Test,Complex1" --singlemethodname Main --singlemethodindex 1
Emitting R2R PE file: c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\\Complex1.dll
I then wanted to see some more detail from the jit. To keep the size of this example small, I'm just using the NgenOrder=1switch, but jit developers would more likely use NgenDump=* switch.
C:\git2\runtime>"dotnet" "c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\Tests\Core_Root\crossgen2\crossgen2.dll" @"c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\\Complex1.dll.rsp" -r:c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\IL-CG2\*.dll --print-repro-instructions --singlemethodtypename "Complex_Array_Test,Complex1" --singlemethodname Main --singlemethodindex 1 --codegenopt NgenOrder=1
C:\git2\runtime\.dotnet
Single method repro args:--singlemethodtypename "Complex_Array_Test,Complex1" --singlemethodname Main --singlemethodindex 1
| Profiled | Method | Method has | calls | Num |LclV |AProp| CSE | Perf |bytes | x64 codesize|
mdToken | CNT | RGN | Hash | EH | FRM | LOOP | NRM | IND | BBs | Cnt | Cnt | Cnt | Score | IL | HOT | CLD | method name
---------+------+------+----------+----+-----+------+-----+-----+-----+-----+-----+-----+---------+------+-------+-----+
06000002 | | | f656934b | | rsp | LOOP | 3 | 0 | 35 | 56 | 48 | 7 | 43056 | 490 | 761 | 0 | Complex_Array_Test:Main(System.String[]):int
Emitting R2R PE file: c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\\Complex1.dll
And finally, as the last --targetarch and --targetos switch is the meaningful one, it is simple to target a different architecture for ad hoc exploration...
C:\git2\runtime>"dotnet" "c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\Tests\Core_Root\crossgen2\crossgen2.dll" @"c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\\Complex1.dll.rsp" -r:c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\IL-CG2\*.dll --print-repro-instructions --singlemethodtypename "Complex_Array_Test,Complex1" --singlemethodname Main --singlemethodindex 1 --codegenopt NgenOrder=1 --targetarch arm64
C:\git2\runtime\.dotnet
Single method repro args:--singlemethodtypename "Complex_Array_Test,Complex1" --singlemethodname Main --singlemethodindex 1
| Profiled | Method | Method has | calls | Num |LclV |AProp| CSE | Perf |bytes | arm64 codesize|
mdToken | CNT | RGN | Hash | EH | FRM | LOOP | NRM | IND | BBs | Cnt | Cnt | Cnt | Score | IL | HOT | CLD | method name
---------+------+------+----------+----+-----+------+-----+-----+-----+-----+-----+-----+---------+------+-------+-----+
06000002 | | | f656934b | | fp | LOOP | 3 | 0 | 35 | 59 | 48 | 10 | 63828 | 490 | 1048 | 0 | Complex_Array_Test:Main(System.String[]):int
Emitting R2R PE file: c:\git2\runtime\artifacts\tests\coreclr\windows.x64.Debug\jit\Directed\Arrays\Complex1\\Complex1.dll
Note that the only difference in the command line was to pass the --targetarch arm64 switch, and the JIT now compiles the method as arm64.
There was a problem hiding this comment.
This is great; I love the example.
btw, I presume you should/must use dotnet.exe to invoke crossgen2, not corerun?
There was a problem hiding this comment.
(I've always found debugging under dotnet.exe to be extremely unfriendly)
There was a problem hiding this comment.
You don' t have to use dotnet.exe. Crossgen2 runs just fine under any recent corerun. For managed debugging (which is what I do most of, dotnet.exe provides a fairly friendly debugging experience that often works better than corerun.) Be aware that in 99-100% of cases, the dotnet.exe in use for this is not a dotnet.exe built locally, but instead the dotnet.exe that is part of the sdk that is used to build the runtime, and that the dotnet.cmd or dotnet.sh in the root of the directory is a simple wrapper which finds the actual dotnet.exe to execute. When debugging, you will want to debug the dotnet.exe itself.
If you would prefer to use a corerun, there is also no particular reason for it to be a corerun associated with the build of the runtime you are developing or debugging. You would just need a corerun that is sufficiently recent that crossgen2 will load and doesn't have a mismatched set of BCL libraries as compared to the runtime crossgen2 was compiled against. It is strongly recommended to use a retail corerun/coreclr.dll for actually running crossgen2.exe, otherwise you will find crossgen2.exe to be extremely slow.
I will add a new bullet point with these details.
|
|
||
| - In the runtime testbed, each test can be commanded to compile with crossgen2 by using environment variables. Just set the `RunCrossgen2` variable to 1, and optionally set the `CompositeBuildMode` variable to 1 if you wish to see the R2R behavior with composite image creation. This is often the easiest way to run a simple test with crossgen2 for developers practiced in the CoreCLR testbed. | ||
|
|
||
| - When building the product the clr.tools subset as well as either the clr.jit or clr.alljits subsets must be built. If the jit interface is changed, the clr.runtime subset must also be rebuilt. |
There was a problem hiding this comment.
I guess it's automatically built with clr.NativeCoreLib after #47216
|
Add a bullet point describing that the version of crossgen2 in the bin directory can be used directly without creating a layout, and the existence of the crossgen2.sln which once the product is built, is an extremely convenient way to debug crossgen2 issues. |
ghost
locked as resolved and limited conversation to collaborators
The jit team has asked me to put together some description of how to debug crossgen2. I've put together this document, and will be discussing it with them on Monday. Please add some comments if you feel there are additional details that would be useful.