Create SuperPMI replay job

[BruceForstall]

Create an Azure DevOps SuperPMI replay job for JIT testing. The job should automatically trigger for JIT pull requests. It should do a superpmi replay on all possible variants (all platforms, including cross-compilers).

Here’s my thinking on how this would work:

1. Build just the JIT (all the JITs, including cross-compiling JITs). Don’t need to build the full coreclr product or libraries. This is the same build that the JIT rolling build AzDO pipeline does. Takes about 4 minutes.
   1. E.g., src\coreclr\build-runtime.cmd x64 checked -skipcrossarchnative -nopgooptimize -skiprestoreoptdata -component alljits
2. Have one job for each flavor of collection: arch/OS. I.e., parallelize over collection flavor. One reason for this is collections are pretty big, so we only copy one collection flavor to a Helix machine.
3. On the AzDO job “build” machine:
   1. Download the spmi collections:
      1. superpmi.py -target_arch x64 -target_os windows -spmi_location …
      2. Same for x64/Linux, x86/windows, arm/windows, arm/Linux, arm64/windows, arm64/Linux
4. Replay only on Windows. We could do Linux, but we want to focus on a Windows-based development environment that our devs use.
5. On the replay machine, iterate over some set of COMPlus variables, including baseline, plus JitStressRegs={1,2,…}, plus JitStress={1,2} (this might give “missing data” errors. JitStress might be unworkable, since there are too many "missing data" cases). The new superpmi.py “-jitoption” flag can be used. For each set chosen:
   1. windows-x64: superpmi.py replay -arch x64 -mch_files <dir with all the MCH files>
   2. windows-x86: superpmi.py replay -arch x86 -mch_files <dir with all the MCH files>
   3. For cross-compiler collections, e.g. Linux/x64:
      1. superpmi.py replay -arch x64 -target_os Linux -mch_files <dir with all the MCH files>
   4. For cross-compiler cross-arch collections, e.g. Linux/arm:
      1. superpmi.py replay -arch x86 -target_arch arm -target_os Linux -mch_files <dir with all the MCH files>
6. Make sure errors are reported properly.

The windows-x64 build has these JITs:

clrjit.dll
clrjit_unix_arm_x64.dll
clrjit_unix_arm64_x64.dll
clrjit_unix_armel_x64.dll
clrjit_unix_osx_arm64_x64.dll
clrjit_unix_x64_x64.dll
clrjit_win_arm_x64.dll
clrjit_win_arm64_x64.dll
clrjit_win_x64_x64.dll
clrjit_win_x86_x64.dll

We probably ignore the osx and armel JITs (we don’t have collections there). And clrjit.dll is the same as clrjit_win_x64_x64.dll, so we don’t need to do both. Thus, we have:

clrjit.dll
clrjit_unix_arm_x64.dll
clrjit_unix_arm64_x64.dll
clrjit_unix_x64_x64.dll
clrjit_win_arm_x64.dll
clrjit_win_arm64_x64.dll
clrjit_win_x86_x64.dll

x86 has (minus armel and clrjit_win_x86_x86.dll which is the same as clrjit.dll):

clrjit.dll
clrjit_unix_arm_x86.dll
clrjit_win_arm_x86.dll

the clrjit_unix_arm_x64.dll and clrjit_unix_arm_x86.dll use the same collection; same for clrjit_win_arm_x64.dll and clrjit_win_arm_x86.dll, and x86 clrjit.dll and x64 clrjit_win_x86_x64.dll.

So we have these collections and replays:

Spmi collection	JIT
-target_arch x64 -target_os windows	X64 clrjit.dll
-target_arch arm -target_os Linux	clrjit_unix_arm_x64.dll, clrjit_unix_arm_x86.dll
-target_arch arm64 -target_os Linux	clrjit_unix_arm64_x64.dll
-target_arch x64 -target_os Linux	clrjit_unix_x64_x64.dll
-target_arch arm -target_os windows	clrjit_win_arm_x64.dll, clrjit_win_arm_x86.dll
-target_arch arm64 -target_os windows	clrjit_win_arm64_x64.dll
-target_arch x86 -target_os windows	clrjit_win_x86_x64.dll, x86 clrjit.dll

Note that the cases where we have multiple JITs that use the same collection, we could either use the same Helix machine where we’ve copied that collection, or we could parallelize each JIT to a separate Helix machine, meaning copying one collection set to multiple Helix machines.

Running “superpmi.py replay” shouldn’t require a Core_Root to be set up. It does require passing “-jit_name” for the cross-compilers, and “-arch x86” to run the x86 version of superpmi.exe (otherwise it will look for the x64 product build).