Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 105 Regressions on 3/21/2023 8:47:40 PM #14569

Closed
performanceautofiler bot opened this issue Mar 28, 2023 · 10 comments
Closed

[Perf] Linux/arm64: 105 Regressions on 3/21/2023 8:47:40 PM #14569

performanceautofiler bot opened this issue Mar 28, 2023 · 10 comments
Assignees
@vargaz
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 7b651743dc951384f6c3d8455c88518022c7c3b9
Compare b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Collections.Sort<IntClass>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Array_Comparison - Duration of single invocation 116.77 μs 124.00 μs 1.06 0.02 False
Array_ComparerStruct - Duration of single invocation 101.79 μs 108.13 μs 1.06 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Sort&lt;IntClass&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Sort&lt;IntClass&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Sort&lt;IntClass&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Sort&lt;IntClass&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<IntClass>.Array_Comparison(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 124.00220313333331 > 121.54082472175.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -22.73335285998016 (T) = (0 -124407.04616782717) / Math.Sqrt((1178046.7691016335 / (16)) + (1276459.8752208282 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.07101017925598267 = (116158.60295020835 - 124407.04616782717) / 116158.60295020835 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;IntClass&gt;.Array_ComparerStruct(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 108.13087601428573 > 106.76445717750002.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -22.43643233887251 (T) = (0 -108706.0077375791) / Math.Sqrt((605920.0092879196 / (16)) + (972738.7085596116 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.06284998437779261 = (102277.8466720467 - 108706.0077375791) / 102277.8466720467 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 7b651743dc951384f6c3d8455c88518022c7c3b9
Compare b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FirstWithPredicate_LastElementMatches - Duration of single invocation 8.85 μs 10.05 μs 1.14 0.02 True
Zip - Duration of single invocation 4.01 μs 4.47 μs 1.11 0.01 False
LastWithPredicate_FirstElementMatches - Duration of single invocation 2.30 μs 2.72 μs 1.18 0.04 False
ToDictionary - Duration of single invocation 2.22 μs 2.44 μs 1.10 0.01 False
SingleWithPredicate_LastElementMatches - Duration of single invocation 1.23 μs 1.45 μs 1.17 0.04 False
WhereSingleOrDefault_LastElementMatches - Duration of single invocation 575.87 ns 819.76 ns 1.42 0.26 False
Select - Duration of single invocation 2.44 μs 2.73 μs 1.12 0.02 False
SingleWithPredicate_LastElementMatches - Duration of single invocation 1.29 μs 1.52 μs 1.17 0.03 False
SelectToArray - Duration of single invocation 2.46 μs 2.78 μs 1.13 0.01 False
Aggregate - Duration of single invocation 1.21 μs 1.54 μs 1.27 0.03 False
SelectToList - Duration of single invocation 1.15 μs 1.26 μs 1.10 0.02 False
SelectToList - Duration of single invocation 2.45 μs 2.66 μs 1.09 0.01 False
SelectToArray - Duration of single invocation 561.84 ns 733.89 ns 1.31 0.29 False
WhereAny_LastElementMatches - Duration of single invocation 638.09 ns 823.46 ns 1.29 0.30 False
SingleWithPredicate_FirstElementMatches - Duration of single invocation 1.22 μs 1.45 μs 1.18 0.03 False
Select - Duration of single invocation 2.44 μs 2.75 μs 1.13 0.02 False
Aggregate_Seed - Duration of single invocation 1.23 μs 1.46 μs 1.19 0.02 False
WhereSingle_LastElementMatches - Duration of single invocation 933.14 ns 1.04 μs 1.12 0.15 False
WhereAny_LastElementMatches - Duration of single invocation 921.43 ns 1.04 μs 1.12 0.10 False
SingleWithPredicate_FirstElementMatches - Duration of single invocation 1.22 μs 1.44 μs 1.18 0.03 False
SelectToList - Duration of single invocation 981.94 ns 1.17 μs 1.19 0.01 True
AnyWithPredicate_LastElementMatches - Duration of single invocation 1.23 μs 1.44 μs 1.17 0.04 False
AnyWithPredicate_LastElementMatches - Duration of single invocation 8.89 μs 10.09 μs 1.14 0.01 True
WhereSingle_LastElementMatches - Duration of single invocation 1.35 μs 1.49 μs 1.11 0.02 False
Where - Duration of single invocation 2.03 μs 2.31 μs 1.14 0.01 False
SelectToArray - Duration of single invocation 834.93 ns 1.11 μs 1.33 0.07 True
Select - Duration of single invocation 1.62 μs 2.03 μs 1.25 0.02 False
Where - Duration of single invocation 1.64 μs 2.10 μs 1.28 0.01 True
FirstWithPredicate_LastElementMatches - Duration of single invocation 1.29 μs 1.51 μs 1.17 0.02 False
LastWithPredicate_FirstElementMatches - Duration of single invocation 1.28 μs 1.46 μs 1.15 0.03 False
WhereLast_LastElementMatches - Duration of single invocation 1.41 μs 1.49 μs 1.06 0.01 False
SingleWithPredicate_LastElementMatches - Duration of single invocation 1.26 μs 1.45 μs 1.15 0.03 False
SelectToArray - Duration of single invocation 665.26 ns 791.60 ns 1.19 0.19 False
SingleWithPredicate_FirstElementMatches - Duration of single invocation 1.27 μs 1.49 μs 1.18 0.03 True
ToDictionary - Duration of single invocation 2.72 μs 2.90 μs 1.07 0.00 False
WhereFirst_LastElementMatches - Duration of single invocation 1.33 μs 1.49 μs 1.12 0.01 False
OrderBy - Duration of single invocation 10.43 μs 11.34 μs 1.09 0.00 True
AnyWithPredicate_LastElementMatches - Duration of single invocation 1.23 μs 1.44 μs 1.17 0.03 False
WhereLast_LastElementMatches - Duration of single invocation 674.25 ns 836.03 ns 1.24 0.27 False
WhereLast_LastElementMatches - Duration of single invocation 970.09 ns 1.06 μs 1.10 0.12 False
LastWithPredicate_FirstElementMatches - Duration of single invocation 20.99 ns 22.70 ns 1.08 0.04 False
All_AllElementsMatch - Duration of single invocation 1.25 μs 1.45 μs 1.16 0.05 False
Select - Duration of single invocation 1.97 μs 2.22 μs 1.13 0.02 False
WhereSingleOrDefault_LastElementMatches - Duration of single invocation 1.34 μs 1.49 μs 1.11 0.03 False
FirstWithPredicate_LastElementMatches - Duration of single invocation 1.26 μs 1.45 μs 1.14 0.01 False
WhereSelect - Duration of single invocation 2.33 μs 2.99 μs 1.29 0.03 False
WhereSingle_LastElementMatches - Duration of single invocation 633.68 ns 824.16 ns 1.30 0.26 False
Where - Duration of single invocation 2.51 μs 2.76 μs 1.10 0.01 False
SelectToArray - Duration of single invocation 1.00 μs 1.14 μs 1.14 0.10 False
AnyWithPredicate_LastElementMatches - Duration of single invocation 1.30 μs 1.52 μs 1.17 0.05 False
SelectToList - Duration of single invocation 715.76 ns 886.00 ns 1.24 0.19 False
WhereAny_LastElementMatches - Duration of single invocation 1.34 μs 1.50 μs 1.12 0.02 False
SelectToList - Duration of single invocation 683.58 ns 841.38 ns 1.23 0.18 False
OrderByDescending - Duration of single invocation 13.75 μs 14.91 μs 1.08 0.01 False
FirstWithPredicate_LastElementMatches - Duration of single invocation 1.24 μs 1.50 μs 1.21 0.04 False
WhereFirst_LastElementMatches - Duration of single invocation 947.99 ns 1.02 μs 1.08 0.14 False
WhereSelect - Duration of single invocation 2.78 μs 3.41 μs 1.23 0.01 False
WhereSelect - Duration of single invocation 1.97 μs 2.72 μs 1.39 0.01 False
WhereFirst_LastElementMatches - Duration of single invocation 667.55 ns 822.13 ns 1.23 0.28 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Linq.Tests.Perf_Enumerable* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Linq.Tests.Perf_Enumerable* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: IOrderedEnumerable)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.052410035875262 > 9.324927305788197.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -92.92482563156791 (T) = (0 -10066.880065030926) / Math.Sqrt((1865.4850494952284 / (16)) + (894.360896502122 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.13090362894514174 = (8901.62504334775 - 10066.880065030926) / 8901.62504334775 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Zip(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.4670883762517875 > 4.224325607564156.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/2/2023 6:22:31 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -20.762161875365447 (T) = (0 -4400.870621915251) / Math.Sqrt((3257.6910192037835 / (16)) + (3216.5611080763942 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.09677654311434333 = (4012.549912326528 - 4400.870621915251) / 4012.549912326528 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.LastWithPredicate_FirstElementMatches(input: IOrderedEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.7175667200674534 > 2.4088083287230875.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -22.941151923631097 (T) = (0 -2643.648173624759) / Math.Sqrt((2464.939704137763 / (16)) + (1371.9743479302624 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.14634458208630863 = (2306.154898742059 - 2643.648173624759) / 2306.154898742059 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.ToDictionary(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4441599531233678 > 2.3187676865289553.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -34.22769187048686 (T) = (0 -2449.9687677836655) / Math.Sqrt((126.66826056845007 / (16)) + (946.4455980123354 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.1107518192467571 = (2205.6851272546933 - 2449.9687677836655) / 2205.6851272546933 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.446417555998945 > 1.2944517983856727.
IsChangePoint: Marked as a change because one of 1/5/2023 11:12:07 AM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -43.77035280110256 (T) = (0 -1450.2228535661554) / Math.Sqrt((106.86861163755583 / (16)) + (357.4391468081155 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.16892823351584704 = (1240.6431909033831 - 1450.2228535661554) / 1240.6431909033831 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSingleOrDefault_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 819.7561811168607 > 635.4658028571946.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -22.045919447838646 (T) = (0 -825.3287135442083) / Math.Sqrt((1418.3407356942746 / (16)) + (24.883899398960438 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.33885957266301314 = (616.441582370447 - 825.3287135442083) / 616.441582370447 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Select(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.734311987756356 > 2.5619849370106467.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -74.08083310145794 (T) = (0 -2756.602831887015) / Math.Sqrt((110.10007506519358 / (16)) + (253.76701177275194 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.13036106653520083 = (2438.692302395547 - 2756.602831887015) / 2438.692302395547 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_LastElementMatches(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.5158246368818145 > 1.3495754937893176.
IsChangePoint: Marked as a change because one of 1/5/2023 11:12:07 AM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -25.952099390718214 (T) = (0 -1541.4179603662421) / Math.Sqrt((85.37473045500634 / (16)) + (1984.6350721755762 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.1969932872814938 = (1287.7415243213063 - 1541.4179603662421) / 1287.7415243213063 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToArray(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.78074239821883 > 2.6206676095157175.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -38.105185173444205 (T) = (0 -2779.753741928338) / Math.Sqrt((643.4056912749274 / (16)) + (422.5667840535666 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.11815365496216569 = (2486.0212454632588 - 2779.753741928338) / 2486.0212454632588 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Aggregate(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.5403227635261196 > 1.2689583757248393.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -31.66882488709 (T) = (0 -1462.7214307995232) / Math.Sqrt((93.41380413659577 / (16)) + (1237.5409636524996 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.20569252446306377 = (1213.179480772615 - 1462.7214307995232) / 1213.179480772615 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToList(input: IList)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2640214176974962 > 1.2021078197432586.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -24.50742437969434 (T) = (0 -1268.0873800446168) / Math.Sqrt((555.5984937785096 / (16)) + (79.79668807465843 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.13595805868299138 = (1116.315316707039 - 1268.0873800446168) / 1116.315316707039 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToList(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.663086059309951 > 2.5661282715491622.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -21.416503253082293 (T) = (0 -2616.5384419873803) / Math.Sqrt((614.0956984824913 / (16)) + (726.2567442066406 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.07429723601823254 = (2435.5814706228784 - 2616.5384419873803) / 2435.5814706228784 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToArray(input: Range)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 733.8916580178593 > 592.0304303072339.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -37.004016488890045 (T) = (0 -729.38145015698) / Math.Sqrt((253.34079972083845 / (16)) + (41.968316150263284 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.2717666400818815 = (573.518306872729 - 729.38145015698) / 573.518306872729 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereAny_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 823.4597707054533 > 667.9628147575714.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -28.710222980588764 (T) = (0 -824.7417209901406) / Math.Sqrt((657.9604383470839 / (16)) + (9.619517230759385 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.28935218984533784 = (639.6558888142666 - 824.7417209901406) / 639.6558888142666 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_FirstElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4460247574952725 > 1.2878291298600972.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 3/1/2023 10:50:04 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -28.30980483828303 (T) = (0 -1466.4616591974132) / Math.Sqrt((58.76652244902434 / (16)) + (1445.3959129621205 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.1915976447392677 = (1230.668477461021 - 1466.4616591974132) / 1230.668477461021 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Select(input: IList)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.7506845582187753 > 2.56110625142588.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -101.77696244385318 (T) = (0 -2756.463632797373) / Math.Sqrt((90.51395193057331 / (16)) + (87.8968430110907 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.1295781920383028 = (2440.2592509540095 - 2756.463632797373) / 2440.2592509540095 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Aggregate_Seed(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4598305927186275 > 1.2855263858702908.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -22.105235025337716 (T) = (0 -1477.9974976393225) / Math.Sqrt((35.20207399315997 / (16)) + (2843.669014236494 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.20697213619134197 = (1224.5498080040306 - 1477.9974976393225) / 1224.5498080040306 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSingle_LastElementMatches(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0417789451427166 > 980.8628004178241.
IsChangePoint: Marked as a change because one of 12/5/2022 9:44:53 PM, 12/21/2022 10:31:08 AM, 1/5/2023 11:12:07 AM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -14.77831413824345 (T) = (0 -1045.4319334861264) / Math.Sqrt((607.0096783041487 / (16)) + (13.209154698751853 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.09619790688502546 = (953.6890436662513 - 1045.4319334861264) / 953.6890436662513 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereAny_LastElementMatches(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0354240032165114 > 982.9056399844732.
IsChangePoint: Marked as a change because one of 12/12/2022 5:25:39 PM, 12/21/2022 10:31:08 AM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -13.343980654591185 (T) = (0 -1037.8271801182473) / Math.Sqrt((732.0960943881321 / (16)) + (5.8604254271515765 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.09556119769139755 = (947.3018780741694 - 1037.8271801182473) / 947.3018780741694 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_FirstElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4435993969759258 > 1.2850650522860694.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -33.5408813137038 (T) = (0 -1455.0650450397816) / Math.Sqrt((55.892864284153546 / (16)) + (951.927363672347 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.1871283895655357 = (1225.7014976891464 - 1455.0650450397816) / 1225.7014976891464 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToList(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1729199817678861 > 1036.4067463779847.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -72.42101480548958 (T) = (0 -1175.128049230829) / Math.Sqrt((57.99991969346828 / (16)) + (69.93205799911411 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.19154101364985518 = (986.2254305718349 - 1175.128049230829) / 986.2254305718349 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4380047027501521 > 1.2882088902237072.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -30.701046425957617 (T) = (0 -1445.1422011369011) / Math.Sqrt((199.65908620370496 / (16)) + (722.5250526762787 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.16688582894575224 = (1238.4606662354838 - 1445.1422011369011) / 1238.4606662354838 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: IOrderedEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.085644041073742 > 9.323104032072136.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -127.14780080823478 (T) = (0 -10086.10981901179) / Math.Sqrt((489.5004891483845 / (16)) + (1328.8575558242503 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.13669091371154368 = (8873.221116968765 - 10086.10981901179) / 8873.221116968765 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSingle_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.493662749184823 > 1.4523217035950133.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/2/2023 4:26:55 AM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -24.204607043703344 (T) = (0 -1498.0395925560558) / Math.Sqrt((450.8913472477829 / (16)) + (49.181594580041626 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.09782765607083253 = (1364.5489656524023 - 1498.0395925560558) / 1364.5489656524023 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Where(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3055927915208403 > 2.1296858437904773.
IsChangePoint: Marked as a change because one of 1/5/2023 11:12:07 AM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -126.5770834365088 (T) = (0 -2308.4665125455317) / Math.Sqrt((37.56255994851081 / (16)) + (57.35959887175386 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.1390213320603225 = (2026.7105167994127 - 2308.4665125455317) / 2026.7105167994127 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToArray(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1144322364503911 > 891.0006615736262.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -64.77511335115791 (T) = (0 -1119.1903956889946) / Math.Sqrt((220.52560299133197 / (16)) + (86.86769078872157 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.32224621755678773 = (846.4311569421659 - 1119.1903956889946) / 846.4311569421659 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Select(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.0282028051096153 > 1.710434132047092.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -149.47775401711934 (T) = (0 -2028.7273951501606) / Math.Sqrt((78.90999111093494 / (16)) + (47.20962178793815 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.24381245110705124 = (1631.0557056608482 - 2028.7273951501606) / 1631.0557056608482 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Where(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.095003624661247 > 1.7245560942660023.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -129.8538865622741 (T) = (0 -2093.1710561764294) / Math.Sqrt((6.022790568362999 / (16)) + (254.5230384290182 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.27293674701551435 = (1644.363760480644 - 2093.1710561764294) / 1644.363760480644 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.5087865276157137 > 1.3422286175414433.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -73.03525635093365 (T) = (0 -1508.8925849348168) / Math.Sqrt((140.78347077078357 / (16)) + (10.532191799440449 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.17292823334875546 = (1286.4321465149408 - 1508.8925849348168) / 1286.4321465149408 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.LastWithPredicate_FirstElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4634701979863665 > 1.3327414112887022.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -28.936366255662833 (T) = (0 -1474.7662173778074) / Math.Sqrt((33.9280546161408 / (16)) + (1071.3774177583891 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.16261988702345448 = (1268.485283830394 - 1474.7662173778074) / 1268.485283830394 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereLast_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4944717164179107 > 1.4506395732221828.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/2/2023 4:26:55 AM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -20.31566731973245 (T) = (0 -1498.7482159836002) / Math.Sqrt((644.5613107155856 / (16)) + (13.435706567154986 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.09491196073759406 = (1368.8298874495438 - 1498.7482159836002) / 1368.8298874495438 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4459344641914347 > 1.3114976225200137.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -24.540312248903398 (T) = (0 -1460.5158067144673) / Math.Sqrt((150.839483382883 / (16)) + (1429.1167682028797 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.16947891613900376 = (1248.860314246889 - 1460.5158067144673) / 1248.860314246889 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToArray(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 791.5969928303663 > 668.012525250627.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -32.67425073622115 (T) = (0 -788.5641717656489) / Math.Sqrt((276.0262908977917 / (16)) + (36.17432673221537 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.21968501175289776 = (646.5310011741033 - 788.5641717656489) / 646.5310011741033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_FirstElementMatches(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4946978131021194 > 1.3373649342009162.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -34.548243367924876 (T) = (0 -1504.3994396930161) / Math.Sqrt((42.522131177687925 / (16)) + (900.1600551798487 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.1786782974466909 = (1276.3443960509987 - 1504.3994396930161) / 1276.3443960509987 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.ToDictionary(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9033684375387674 > 2.8468205388766914.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -45.763071863754526 (T) = (0 -2886.0686969713643) / Math.Sqrt((163.77604525738343 / (16)) + (84.69593659463402 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.0632770126591144 = (2714.3149551909246 - 2886.0686969713643) / 2714.3149551909246 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereFirst_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4919797565273105 > 1.4217673730805538.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -35.189429175200715 (T) = (0 -1482.8846059420935) / Math.Sqrt((225.21257305606682 / (16)) + (40.15904060602987 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.10451839289022008 = (1342.562165996886 - 1482.8846059420935) / 1342.562165996886 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.OrderBy(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.337952283187985 > 10.955803151310864.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -35.52244449830977 (T) = (0 -11351.330994452323) / Math.Sqrt((9784.52548411469 / (16)) + (230.56976767661547 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.08465392094173597 = (10465.39433019952 - 11351.330994452323) / 10465.39433019952 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4404604484377903 > 1.292013811716515.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/2/2023 6:22:31 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -63.378290668614945 (T) = (0 -1444.1082188492046) / Math.Sqrt((154.0083601073867 / (16)) + (17.07311867381132 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.1648815525014751 = (1239.7039130271367 - 1444.1082188492046) / 1239.7039130271367 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereLast_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 836.0282378264959 > 674.8307548926956.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -26.750668843526388 (T) = (0 -829.5087882887049) / Math.Sqrt((751.7278831342056 / (16)) + (17.50381607095426 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.2868530006181295 = (644.6025986575444 - 829.5087882887049) / 644.6025986575444 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereLast_LastElementMatches(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.063940956387984 > 984.6906650524363.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -21.676924327202695 (T) = (0 -1050.9546165004647) / Math.Sqrt((319.96382887349904 / (16)) + (55.30395161741314 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.1084785144857303 = (948.10553634235 - 1050.9546165004647) / 948.10553634235 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.LastWithPredicate_FirstElementMatches(input: IList)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.69938570247343 > 22.030211974745043.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -21.69476992164866 (T) = (0 -22.627238870785867) / Math.Sqrt((0.08937477161835572 / (16)) + (0.00551238767853799 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.07903800382389156 = (20.969825706415833 - 22.627238870785867) / 20.969825706415833 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.All_AllElementsMatch(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4461508863993708 > 1.3079518822467997.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -21.41563101477121 (T) = (0 -1473.4525404489957) / Math.Sqrt((28.227246529317185 / (16)) + (2449.979147593429 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.18285615549016537 = (1245.6734773793435 - 1473.4525404489957) / 1245.6734773793435 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Select(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.2206766600727113 > 2.055039065730374.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -88.51174577208042 (T) = (0 -2218.730459675185) / Math.Sqrt((122.9547960448728 / (16)) + (28.24880346724165 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.13568054123429787 = (1953.6571941822567 - 2218.730459675185) / 1953.6571941822567 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSingleOrDefault_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4909163490122745 > 1.41615418466427.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -61.2878199957879 (T) = (0 -1496.3561602216062) / Math.Sqrt((58.339822156240885 / (16)) + (48.35120788237667 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.10989473198386943 = (1348.1964704409042 - 1496.3561602216062) / 1348.1964704409042 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4466682436423013 > 1.2951890456629573.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -33.290623280250024 (T) = (0 -1450.2037679860152) / Math.Sqrt((362.30851208045874 / (16)) + (358.72629487416253 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.16722505876014304 = (1242.4371436357437 - 1450.2037679860152) / 1242.4371436357437 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSelect(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9949860530678802 > 2.4601736571324353.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -132.81246142002885 (T) = (0 -2996.507094546279) / Math.Sqrt((285.93335319244665 / (16)) + (142.3741496533959 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.27988204762324126 = (2341.236913284966 - 2996.507094546279) / 2341.236913284966 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSingle_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 824.1602368978295 > 666.499031219402.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -20.807332740828578 (T) = (0 -828.5078342929263) / Math.Sqrt((1435.4887028975706 / (16)) + (18.89109973276811 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.3140891572398278 = (630.4806867390654 - 828.5078342929263) / 630.4806867390654 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Where(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.75911712299481 > 2.614374753053478.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -50.15407187902612 (T) = (0 -2766.1106542038783) / Math.Sqrt((131.1112629775574 / (16)) + (476.8639053205757 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.10999583314136532 = (2492.000935152696 - 2766.1106542038783) / 2492.000935152696 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToArray(input: IList)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1440715286248488 > 1.054700334644758.
IsChangePoint: Marked as a change because one of 12/14/2022 12:26:48 AM, 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -26.32398762190168 (T) = (0 -1151.1606207875145) / Math.Sqrt((596.2725932288328 / (16)) + (83.70731062458378 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.17171591918540832 = (982.4570972696316 - 1151.1606207875145) / 982.4570972696316 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.515382817312103 > 1.3674280242944825.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -42.100389514278376 (T) = (0 -1529.4382423688182) / Math.Sqrt((9.354223570764452 / (16)) + (632.2821041570147 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.17517577351273816 = (1301.4548775092155 - 1529.4382423688182) / 1301.4548775092155 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToList(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 886.0036569851969 > 755.9782540401552.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -26.398807396736593 (T) = (0 -889.2765986714295) / Math.Sqrt((528.9368191352838 / (16)) + (36.61261421441917 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.2120126185198104 = (733.7189275780582 - 889.2765986714295) / 733.7189275780582 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereAny_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.49784277630095 > 1.4117671772551161.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -41.173959176998984 (T) = (0 -1492.0999528042316) / Math.Sqrt((163.4861510339554 / (16)) + (31.144752338786372 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.1038987118708523 = (1351.663822739197 - 1492.0999528042316) / 1351.663822739197 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToList(input: Range)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 841.3840261876205 > 715.0552193476791.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -19.081838667470077 (T) = (0 -848.6744154901481) / Math.Sqrt((957.3656595284683 / (16)) + (63.837248924635084 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.2166792034681509 = (697.5334279331782 - 848.6744154901481) / 697.5334279331782 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.OrderByDescending(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.90844110951926 > 14.39909631682552.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -18.849275849749887 (T) = (0 -14926.350880425762) / Math.Sqrt((56395.542609103686 / (16)) + (7324.8570454229675 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.08510929939233301 = (13755.619723086511 - 14926.350880425762) / 13755.619723086511 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.5038328042016447 > 1.2962454789521443.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -20.3285446580566 (T) = (0 -1455.3669427933967) / Math.Sqrt((663.4277955156423 / (16)) + (1480.9329230319895 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.17052485631744593 = (1243.3456111065288 - 1455.3669427933967) / 1243.3456111065288 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereFirst_LastElementMatches(input: List)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.023915190547527 > 960.4526086479152.
IsChangePoint: Marked as a change because one of 12/5/2022 9:44:53 PM, 12/21/2022 10:31:08 AM, 1/5/2023 11:12:07 AM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -15.034378358375708 (T) = (0 -1028.4960500237137) / Math.Sqrt((603.8846718359838 / (16)) + (15.84513227539535 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.09969586849866585 = (935.254991389432 - 1028.4960500237137) / 935.254991389432 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSelect(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.410527000621481 > 2.9366513083476184.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -52.50269333827978 (T) = (0 -3434.5103091891665) / Math.Sqrt((477.43841381598776 / (16)) + (2489.65933350309 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.22370202945668102 = (2806.655727059696 - 3434.5103091891665) / 2806.655727059696 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSelect(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.723899344569288 > 2.0576467068227497.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -170.68535189918262 (T) = (0 -2730.176188422132) / Math.Sqrt((260.8732285941371 / (16)) + (90.25352566890709 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.3935726081711878 = (1959.1201580841882 - 2730.176188422132) / 1959.1201580841882 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereFirst_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 822.1330679719212 > 693.6961632964802.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -18.93446433644892 (T) = (0 -820.9538124160122) / Math.Sqrt((1502.726862956322 / (16)) + (30.38674498090401 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.2905832019706815 = (636.1107220072604 - 820.9538124160122) / 636.1107220072604 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 7b651743dc951384f6c3d8455c88518022c7c3b9
Compare b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Collections.Sort<IntStruct>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LinqQuery - Duration of single invocation 118.47 μs 132.87 μs 1.12 0.00 True
Array_ComparerClass - Duration of single invocation 50.11 μs 64.02 μs 1.28 0.00 True
LinqOrderByExtension - Duration of single invocation 118.82 μs 132.68 μs 1.12 0.00 True
Array_Comparison - Duration of single invocation 48.65 μs 63.89 μs 1.31 0.00 True
Array_ComparerStruct - Duration of single invocation 56.72 μs 67.26 μs 1.19 0.06 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Sort&lt;IntStruct&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Sort&lt;IntStruct&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Sort&lt;IntStruct&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Sort&lt;IntStruct&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<IntStruct>.LinqQuery(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 132.87367225333335 > 124.404778911.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -93.88892041117951 (T) = (0 -132529.4850592641) / Math.Sqrt((231904.29922198324 / (16)) + (163727.09842165175 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.1172257649691981 = (118623.7278218498 - 132529.4850592641) / 118623.7278218498 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Collections.Sort&lt;IntStruct&gt;.Array_ComparerClass(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 64.02376672 > 52.18673502753846.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -143.0731621574035 (T) = (0 -64352.96528748917) / Math.Sqrt((57551.625207608406 / (16)) + (149175.93353637794 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.2928003982011585 = (49777.959054647436 - 64352.96528748917) / 49777.959054647436 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Collections.Sort&lt;IntStruct&gt;.LinqOrderByExtension(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 132.6798791692308 > 124.34412784550001.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -76.49813334191317 (T) = (0 -132745.62116179653) / Math.Sqrt((270742.41015772335 / (16)) + (390958.194512871 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.12024274051791035 = (118497.19383178114 - 132745.62116179653) / 118497.19383178114 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Collections.Sort&lt;IntStruct&gt;.Array_Comparison(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.8937141142857 > 51.115523697.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -272.842347956115 (T) = (0 -63667.00674447885) / Math.Sqrt((15467.873323355385 / (16)) + (44342.417014517065 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.3055359505566963 = (48766.950245476175 - 63667.00674447885) / 48766.950245476175 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Collections.Sort&lt;IntStruct&gt;.Array_ComparerStruct(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 67.26443017333332 > 59.56151669550001.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -36.43107576401793 (T) = (0 -66853.08627956711) / Math.Sqrt((1111474.5389564182 / (16)) + (216684.79818394067 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.18129734524769636 = (56592.93703529762 - 66853.08627956711) / 56592.93703529762 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 7b651743dc951384f6c3d8455c88518022c7c3b9
Compare b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_VectorOf<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
MaxBenchmark - Duration of single invocation 26.01 ns 27.63 ns 1.06 0.03 False
NegateBenchmark - Duration of single invocation 0.95 ns 29.31 ns 30.83 0.06 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 1.30 ns 9.64 ns 7.43 0.09 False
BitwiseAndBenchmark - Duration of single invocation 1.16 ns 9.45 ns 8.16 0.34 False
OneBenchmark - Duration of single invocation 0.69 ns 10.99 ns 15.96 0.32 False
SubtractBenchmark - Duration of single invocation 1.29 ns 29.56 ns 22.84 0.06 False
AddBenchmark - Duration of single invocation 1.31 ns 29.19 ns 22.27 0.06 False
OnesComplementBenchmark - Duration of single invocation 0.94 ns 8.89 ns 9.42 0.13 False
UnaryNegateOperatorBenchmark - Duration of single invocation 0.89 ns 29.23 ns 32.84 0.05 False
BitwiseOrOperatorBenchmark - Duration of single invocation 1.25 ns 9.64 ns 7.74 0.11 False
XorBenchmark - Duration of single invocation 1.26 ns 9.56 ns 7.58 0.34 False
GreaterThanBenchmark - Duration of single invocation 24.14 ns 27.63 ns 1.14 0.01 True
BitwiseOrBenchmark - Duration of single invocation 1.23 ns 9.65 ns 7.84 0.34 False
GreaterThanOrEqualBenchmark - Duration of single invocation 25.13 ns 28.74 ns 1.14 0.01 False
SubtractionOperatorBenchmark - Duration of single invocation 1.31 ns 29.43 ns 22.42 0.07 False
EqualsStaticBenchmark - Duration of single invocation 24.15 ns 27.88 ns 1.15 0.01 True
ConditionalSelectBenchmark - Duration of single invocation 1.92 ns 32.80 ns 17.12 0.35 False
AndNotBenchmark - Duration of single invocation 1.31 ns 18.29 ns 13.99 0.07 False
BitwiseAndOperatorBenchmark - Duration of single invocation 1.32 ns 9.57 ns 7.23 0.06 False
MinBenchmark - Duration of single invocation 26.01 ns 27.76 ns 1.07 0.01 False
AddOperatorBenchmark - Duration of single invocation 1.30 ns 29.18 ns 22.46 0.08 False
OnesComplementOperatorBenchmark - Duration of single invocation 0.92 ns 9.16 ns 9.98 0.09 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Single>.MaxBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.627589375832873 > 27.384542164172853.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -45.05121386873301 (T) = (0 -27.762382781028816) / Math.Sqrt((0.0026174811122003806 / (16)) + (0.026997319115554412 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.06441456491241464 = (26.082302606704037 - 27.762382781028816) / 26.082302606704037 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.309969717306544 > 0.9743768838128409.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -836.6233571232846 (T) = (0 -29.34436525975851) / Math.Sqrt((0.011844029434336982 / (27)) + (0.0078226330659942 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -29.085862224935887 = (0.9753539732505054 - 29.34436525975851) / 0.9753539732505054 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.636478812158922 > 1.3052379695203482.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -202.30737415082484 (T) = (0 -9.604305707077746) / Math.Sqrt((0.040789205620951556 / (27)) + (0.0023275306116026465 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.947849855652826 = (1.208415594344272 - 9.604305707077746) / 1.208415594344272 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.445932020755402 > 1.206662503189234.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -48.158340845888276 (T) = (0 -9.776901881214062) / Math.Sqrt((0.027217787242668497 / (27)) + (0.33963970787301745 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.302042796177023 = (1.1776501424102743 - 9.776901881214062) / 1.1776501424102743 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.OneBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.987079935458715 > 0.8436627177094524.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -194.51572060383782 (T) = (0 -11.03229598944712) / Math.Sqrt((0.04570686666055736 / (27)) + (0.010587738353400671 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -9.93553119115248 = (1.0088486600790758 - 11.03229598944712) / 1.0088486600790758 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.559583423513192 > 1.2119499049527773.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -345.4603233748499 (T) = (0 -29.1508255918785) / Math.Sqrt((0.1214452413845723 / (27)) + (0.022602452827779528 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -23.575828079191762 = (1.1861584276202015 - 29.1508255918785) / 1.1861584276202015 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.186159094630906 > 1.2244407268626105.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -155.4791839609395 (T) = (0 -29.291572279142596) / Math.Sqrt((0.0195669328700541 / (27)) + (0.3525454744513875 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -24.601554140747727 = (1.1441325834403855 - 29.291572279142596) / 1.1441325834403855 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.890385677390688 > 0.9848021119210807.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -177.62857317867517 (T) = (0 -9.07246672918129) / Math.Sqrt((0.022357237178207517 / (27)) + (0.013579481311540959 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.023538797563676 = (1.0054222553607044 - 9.07246672918129) / 1.0054222553607044 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.22812991146024 > 0.9545947307863333.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -708.7209428035404 (T) = (0 -29.329610425171655) / Math.Sqrt((0.01416171555684798 / (27)) + (0.011829146190533496 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -28.87353840101751 = (0.9817923150400112 - 29.329610425171655) / 0.9817923150400112 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.642079010360822 > 1.1790222601466427.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -50.357859987042616 (T) = (0 -9.7379297523181) / Math.Sqrt((0.10860615057241876 / (27)) + (0.27333143902571455 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -7.242528714091672 = (1.1814250323046915 - 9.7379297523181) / 1.1814250323046915 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.563390762318214 > 1.2577005758531516.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -165.30918759054288 (T) = (0 -9.583555403195827) / Math.Sqrt((0.05913611557397001 / (27)) + (0.0041992623508128315 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.986597565908971 = (1.199954714646387 - 9.583555403195827) / 1.199954714646387 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.GreaterThanBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.633185812874206 > 25.344653720179057.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -46.247910620854505 (T) = (0 -27.698845470250852) / Math.Sqrt((0.052021595296725724 / (16)) + (0.04782275776054384 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.14022495987264885 = (24.29243916335994 - 27.698845470250852) / 24.29243916335994 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.652005622205497 > 1.3685988019001443.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -125.89184170672809 (T) = (0 -9.588139729940965) / Math.Sqrt((0.09625239818324106 / (27)) + (0.008906611955021081 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.600213305199109 = (1.2615619252925396 - 9.588139729940965) / 1.2615619252925396 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.GreaterThanOrEqualBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.73568395877691 > 26.412730936193412.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -52.6307867094083 (T) = (0 -28.8752677029041) / Math.Sqrt((0.03991058640185439 / (16)) + (0.048467136413092984 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.14275775806762162 = (25.268056592966428 - 28.8752677029041) / 25.268056592966428 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.43213370277526 > 1.373733827898184.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -607.6677197836968 (T) = (0 -29.19864434821344) / Math.Sqrt((0.024933327106635282 / (27)) + (0.013170685588490128 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -23.045696564403613 = (1.214298128981552 - 29.19864434821344) / 1.214298128981552 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.EqualsStaticBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.88497265690818 > 25.377411978756406.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -50.7493278680416 (T) = (0 -27.72215811478601) / Math.Sqrt((0.042697373419776925 / (16)) + (0.04387974531745419 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.14286899587041443 = (24.256636775479837 - 27.72215811478601) / 24.256636775479837 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.80411017761427 > 1.9058304061821347.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -365.1314974855084 (T) = (0 -32.85263349617441) / Math.Sqrt((0.16862638752649223 / (27)) + (0.012280198222831223 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -20.556799526059738 = (1.5240032944806712 - 32.85263349617441) / 1.5240032944806712 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.288451568319147 > 1.3782974481056427.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -192.44444876440193 (T) = (0 -17.949133826513698) / Math.Sqrt((0.051276129913315635 / (27)) + (0.062009767297654825 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -13.444850583505078 = (1.2425974033272622 - 17.949133826513698) / 1.2425974033272622 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.567155820992202 > 1.3768822755355905.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -176.13786667850147 (T) = (0 -9.612917562375209) / Math.Sqrt((0.056870491859839944 / (27)) + (0.001799917130304443 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -6.872866732746934 = (1.2210187075046741 - 9.612917562375209) / 1.2210187075046741 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.MinBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.759036759548533 > 27.301065053009093.
IsChangePoint: Marked as a change because one of 3/6/2023 9:06:14 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -39.20470567350574 (T) = (0 -27.80146016063493) / Math.Sqrt((0.0032635223403330634 / (16)) + (0.0412617438559769 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.0687252853886387 = (26.013663698922418 - 27.80146016063493) / 26.013663698922418 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.17660490230657 > 1.2013005207558176.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -259.0178321990306 (T) = (0 -29.254772186581473) / Math.Sqrt((0.04380938986005523 / (27)) + (0.11123289505735398 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -23.455355672190574 = (1.1962521657310656 - 29.254772186581473) / 1.1962521657310656 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.160725462472241 > 0.9635481956558107.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -308.78359557260984 (T) = (0 -9.131739053772087) / Math.Sqrt((0.004395502218493079 / (27)) + (0.005923149798702022 / (11))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (11) - 2, .025) and -8.564863132897585 = (0.9547171691735136 - 9.131739053772087) / 0.9547171691735136 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 7b651743dc951384f6c3d8455c88518022c7c3b9
Compare b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Tests.Perf_TimeSpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FromSeconds - Duration of single invocation 12.85 ns 14.79 ns 1.15 0.05 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_TimeSpan*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_TimeSpan* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_TimeSpan*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_TimeSpan* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_TimeSpan.FromSeconds

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.786134903854672 > 13.38753834142601.
IsChangePoint: Marked as a change because one of 12/14/2022 12:01:27 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -71.8817903047114 (T) = (0 -14.875088670963947) / Math.Sqrt((0.002462879081091379 / (16)) + (0.01613990712101692 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.16817733602152404 = (12.733587797229674 - 14.875088670963947) / 12.733587797229674 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 7b651743dc951384f6c3d8455c88518022c7c3b9
Compare b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in LinqBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Where00LinqQueryX - Duration of single invocation 1.63 secs 2.07 secs 1.27 0.15 False
Where00LinqMethodX - Duration of single invocation 1.60 secs 2.07 secs 1.30 0.17 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'LinqBenchmarks*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter LinqBenchmarks* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'LinqBenchmarks*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter LinqBenchmarks* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

LinqBenchmarks.Where00LinqQueryX

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.0664345357857146 > 1.6982661364067306.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -81.71614078661506 (T) = (0 -2054776509.5215452) / Math.Sqrt((215720030542943.44 / (16)) + (346883266373803.3 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.27401968164477 = (1612829486.9580128 - 2054776509.5215452) / 1612829486.9580128 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### LinqBenchmarks.Where00LinqMethodX

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.072636478466667 > 1.6826957343807691.
IsChangePoint: Marked as a change because one of 1/5/2023 11:12:07 AM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -79.00194921031215 (T) = (0 -2055177809.4761903) / Math.Sqrt((201487486955033.16 / (16)) + (403247448639429.8 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.27187511150366095 = (1615864475.1263967 - 2055177809.4761903) / 1615864475.1263967 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 7b651743dc951384f6c3d8455c88518022c7c3b9
Compare b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Memory.Constructors<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadOnlySpanImplicitCastFromArray - Duration of single invocation 4.93 ns 6.61 ns 1.34 0.11 False
ReadOnlySpanFromArray - Duration of single invocation 4.91 ns 6.60 ns 1.34 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Constructors&lt;String&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.Constructors&lt;String&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Constructors&lt;String&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.Constructors&lt;String&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Memory.Constructors<String>.ReadOnlySpanImplicitCastFromArray

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.6115812956439255 > 5.178969486868454.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -14.731153783685157 (T) = (0 -6.619186712081047) / Math.Sqrt((0.2679734193684743 / (16)) + (0.00541569824731726 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.4087003861967695 = (4.698789591413137 - 6.619186712081047) / 4.698789591413137 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Constructors&lt;String&gt;.ReadOnlySpanFromArray

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.601084084162827 > 5.160680233842701.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -96.18411661187609 (T) = (0 -6.622411129260646) / Math.Sqrt((0.001214340679620057 / (16)) + (0.0051995151157671156 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.3452492898801345 = (4.92281332469666 - 6.622411129260646) / 4.92281332469666 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 7b651743dc951384f6c3d8455c88518022c7c3b9
Compare b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Collections.Sort<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LinqOrderByExtension - Duration of single invocation 100.76 μs 109.28 μs 1.08 0.04 False
Array_ComparerStruct - Duration of single invocation 50.62 μs 56.19 μs 1.11 0.15 False
Array_ComparerClass - Duration of single invocation 43.56 μs 53.58 μs 1.23 0.00 True
LinqQuery - Duration of single invocation 101.26 μs 109.60 μs 1.08 0.01 True
Array_Comparison - Duration of single invocation 42.92 μs 52.72 μs 1.23 0.00 True

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Sort&lt;Int32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Sort&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Sort&lt;Int32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Sort&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<Int32>.LinqOrderByExtension(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 109.27922692000001 > 105.98860161992309.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -82.13144889223777 (T) = (0 -109635.31936716619) / Math.Sqrt((95353.42400318095 / (16)) + (121012.97230944711 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.08718857334252123 = (100842.96510778846 - 109635.31936716619) / 100842.96510778846 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;Int32&gt;.Array_ComparerStruct(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 56.19317819999999 > 54.04066856946153.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -12.614848174324669 (T) = (0 -56112.35808961038) / Math.Sqrt((2465513.8513712958 / (16)) + (137915.37033677837 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.0989345777276666 = (51060.690260231226 - 56112.35808961038) / 51060.690260231226 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;Int32&gt;.Array_ComparerClass(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 53.57966364 > 46.053349657.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -134.92221428166602 (T) = (0 -53544.57467011322) / Math.Sqrt((58712.96408387623 / (16)) + (33930.67459945524 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.22236436466355833 = (43804.10311196429 - 53544.57467011322) / 43804.10311196429 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Collections.Sort&lt;Int32&gt;.LinqQuery(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 109.59815322666667 > 105.3687359795.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -62.3508069738072 (T) = (0 -109719.65956422077) / Math.Sqrt((205886.10022647667 / (16)) + (186746.80219130238 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.0905678502308488 = (100607.82512614467 - 109719.65956422077) / 100607.82512614467 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Collections.Sort&lt;Int32&gt;.Array_Comparison(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 52.71822948 > 44.86096784950001.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -149.3143298180312 (T) = (0 -52784.670962380966) / Math.Sqrt((63267.50988958866 / (16)) + (10977.060921555098 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.23269316170082782 = (42820.60824410714 - 52784.670962380966) / 42820.60824410714 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 28, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 7b651743dc951384f6c3d8455c88518022c7c3b9
Compare b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InstanceDelegate - Duration of single invocation 400.95 μs 570.36 μs 1.42 0.34 False
MeasureEvents - Duration of single invocation 49.72 ms 73.40 ms 1.48 0.16 True

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.LowLevelPerf*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.LowLevelPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.LowLevelPerf*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.LowLevelPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.InstanceDelegate

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 570.3556020833333 > 421.0268955806213.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -46.25494134954574 (T) = (0 -566525.2910413162) / Math.Sqrt((128935616.59747888 / (16)) + (90500489.34120655 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.398315719062724 = (405148.3390468156 - 566525.2910413162) / 405148.3390468156 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.LowLevelPerf.MeasureEvents

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 73.39619367857144 > 53.23996327125.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -53.91436172308612 (T) = (0 -73246468.88544375) / Math.Sqrt((2658328967447.7744 / (16)) + (194475145966.6614 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.44489110904493245 = (50693417.95165408 - 73246468.88544375) / 50693417.95165408 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 7b651743dc951384f6c3d8455c88518022c7c3b9
Compare b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Collections.Sort<BigStruct>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Array_ComparerClass - Duration of single invocation 76.76 μs 83.05 μs 1.08 0.00 True
LinqOrderByExtension - Duration of single invocation 160.04 μs 169.89 μs 1.06 0.00 False
Array_Comparison - Duration of single invocation 75.78 μs 81.87 μs 1.08 0.00 True
Array_ComparerStruct - Duration of single invocation 79.91 μs 86.51 μs 1.08 0.00 True

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Sort&lt;BigStruct&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Sort&lt;BigStruct&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Sort&lt;BigStruct&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Sort&lt;BigStruct&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<BigStruct>.Array_ComparerClass(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 83.05390535714288 > 80.54140767884613.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -42.448236317632336 (T) = (0 -83137.69466359307) / Math.Sqrt((350129.23645504453 / (16)) + (137351.07605082035 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.09364731450618004 = (76018.74348416668 - 83137.69466359307) / 76018.74348416668 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Collections.Sort&lt;BigStruct&gt;.LinqOrderByExtension(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 169.89184281333334 > 167.9217660895.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -27.534791989789785 (T) = (0 -170508.3701708525) / Math.Sqrt((1109962.1435916321 / (16)) + (1464245.9510247144 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.06330692644165575 = (160356.68152887592 - 170508.3701708525) / 160356.68152887592 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;BigStruct&gt;.Array_Comparison(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 81.86923314285717 > 79.62845063538461.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -42.14670728902827 (T) = (0 -81816.97036624042) / Math.Sqrt((166400.71352462407 / (16)) + (245052.34252159324 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.08178462915361903 = (75631.47798675367 - 81816.97036624042) / 75631.47798675367 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Collections.Sort&lt;BigStruct&gt;.Array_ComparerStruct(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 86.51300261428571 > 83.84426619950001.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -58.37613173269117 (T) = (0 -86839.23074907094) / Math.Sqrt((103572.45065423827 / (16)) + (175873.78305693064 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.08796978737668629 = (79817.68589223214 - 86839.23074907094) / 79817.68589223214 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 7b651743dc951384f6c3d8455c88518022c7c3b9
Compare b0b7aaefb88aa8d01b3d64fb40ac2f73a9d98c3e
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Concat_CharEnumerable - Duration of single invocation 17.73 μs 20.08 μs 1.13 0.01 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_String*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_String* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_String*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_String* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Concat_CharEnumerable

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.082623317307696 > 18.63794147790274.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 3/28/2023 2:42:21 AM falls between 3/19/2023 2:30:51 PM and 3/28/2023 2:42:21 AM.
IsRegressionStdDev: Marked as regression because -96.19241504234249 (T) = (0 -20195.32804814706) / Math.Sqrt((591.3206508229594 / (16)) + (13342.164190126347 / (22))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (22) - 2, .025) and -0.13742286437189283 = (17755.33856469407 - 20195.32804814706) / 17755.33856469407 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kotlarmilos
Copy link
Member

@vargaz could mono_memory_barrier in dotnet/runtime#83673 cause the regressions?

@kotlarmilos kotlarmilos added LinkedFromMaster Is this issue linked from the master perf issue and removed untriaged labels Mar 28, 2023
@kotlarmilos kotlarmilos mentioned this issue Mar 28, 2023
Closed
@vargaz
Copy link

vargaz commented Mar 28, 2023

Possible, will investigate.

@BrzVlad
Copy link
Member

BrzVlad commented Jun 30, 2023

arm64 has load acquire instruction, maybe using it instead of a full membar dmb will significantly reduce the regressions here

@vargaz
Copy link

vargaz commented Jul 2, 2023

Benchmarking on an apple m2:

  • The dmb instruction doesn't seem to cause a slowdown
  • Acquire loads don't seem to work
  • Adding an indirection solves the original race, but leads to about a 5-10% slowdown

@kotlarmilos
Copy link
Member

newplot (4)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vargaz vargaz

Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vargaz @BrzVlad @kotlarmilos