JIT: Libraries PGO assert during find loops

[AndyAyersMS]

This is from windows arm64; it may fail on other os/arch combos too.

COMPlus_JitRandomEdgeCounts=1
COMPlus_JitRandomGuardedDevirtualization=1
COMPlus_ReadyToRun=0
COMPlus_TC_QuickJitForLoops=1
COMPlus_TieredCompilation=1
COMPlus_TieredPGO=1

dotnet.exe exec --runtimeconfig System.Globalization.Nls.Tests.runtimeconfig.json --depsfile System.Globalization.Nls.Tests.deps.json xunit.console.dll System.Globalization.Nls.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Globalization.Nls.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Globalization.Nls.Tests (found 443 of 447 test cases)
  Starting:    System.Globalization.Nls.Tests (parallel test collections = on, max threads = 8)
    ...

Assert failure(PID 11056 [0x00002b30], Thread: 6520 [0x1978]): Assertion failed 'optLoopTable[i].lpContainedBy(first, bottom)' in 'System.String:SplitInternal(System.ReadOnlySpan`1[Char],int,int):System.String[]:this' during 'Find loops' (IL size 224)

    File: D:\workspace\_work\1\s\src\coreclr\jit\optimizer.cpp Line: 1099
    Image: D:\h\w\B21709C2\p\dotnet.exe

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This is from windows arm64; it may fail on other os/arch combos too.

COMPlus_JitRandomEdgeCounts=1
COMPlus_JitRandomGuardedDevirtualization=1
COMPlus_ReadyToRun=0
COMPlus_TC_QuickJitForLoops=1
COMPlus_TieredCompilation=1
COMPlus_TieredPGO=1

dotnet.exe exec --runtimeconfig System.Globalization.Nls.Tests.runtimeconfig.json --depsfile System.Globalization.Nls.Tests.deps.json xunit.console.dll System.Globalization.Nls.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Globalization.Nls.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Globalization.Nls.Tests (found 443 of 447 test cases)
  Starting:    System.Globalization.Nls.Tests (parallel test collections = on, max threads = 8)
    ...

Assert failure(PID 11056 [0x00002b30], Thread: 6520 [0x1978]): Assertion failed 'optLoopTable[i].lpContainedBy(first, bottom)' in 'System.String:SplitInternal(System.ReadOnlySpan`1[Char],int,int):System.String[]:this' during 'Find loops' (IL size 224)

    File: D:\workspace\_work\1\s\src\coreclr\jit\optimizer.cpp Line: 1099
    Image: D:\h\w\B21709C2\p\dotnet.exe

Author:	AndyAyersMS
Assignees:	-
Labels:	area-CodeGen-coreclr, untriaged
Milestone:	-

[AndyAyersMS]

cc @dotnet/jit-contrib

[BruceForstall]

@AndyAyersMS A side comment: the repro has randomness from (1) JitRandomEdgeCounts, (2) JitRandomGuardedDevirtualization, (3) Tiered compilation, (4) Tiered PGO.

Possibly (1) and (2) aren't too random due to a hash-based random number seed? And (4) is not actually random? So the real randomness is from (3)?

A question is: can we collect "side data" from the randomness on a win/arm64 run that we can replay using an altjit on win/x64 to reproduce the issue? Presumably, we could do this using the existing record/replay mechanism of SuperPMI today already, if we implemented #41649?

Regarding this issue: did it come from the experimental pipeline?

[AndyAyersMS]

It's from our regular weekend CI runs:

https://dev.azure.com/dnceng/public/_build?definitionId=1002&_a=summary

Yes, true randomness here is likely from variability in what gets tiered up, for libraries runs (given high level of concurrency and load) this can fluctuate quite a bit from run to run. And of course changes in libraries code (eg Regex was revised recently to support source generation...) can alter things too.

The side data collection would be nice addition -- the idea I had was to always be collecting SPMI data for these runs but (on the fly) toss out the MC's for any method that jits successfully. Not sure how feasible this is.

[AndyAyersMS]

Failed again over the weekend:

https://dev.azure.com/dnceng/public/_build/results?buildId=1400396&view=ms.vss-test-web.build-test-results-tab

[AndyAyersMS]

Suspect this was fixed by #69878. Haven't seen it happening recently.