Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 315 Improvements on 3/1/2023 1:48:25 AM #13761

Closed
performanceautofiler bot opened this issue Mar 8, 2023 · 23 comments
Closed

[Perf] Linux/arm64: 315 Improvements on 3/1/2023 1:48:25 AM #13761

performanceautofiler bot opened this issue Mar 8, 2023 · 23 comments
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-improvement runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Reader.Tests.Perf_Base64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadBase64EncodedByteArray_NoEscaping - Duration of single invocation 4.40 μs 3.74 μs 0.85 0.03 False
ReadBase64EncodedByteArray_HeavyEscaping - Duration of single invocation 4.39 μs 3.66 μs 0.83 0.03 False
ReadBase64EncodedByteArray_NoEscaping - Duration of single invocation 34.27 μs 26.24 μs 0.77 0.09 False
ReadBase64EncodedByteArray_HeavyEscaping - Duration of single invocation 34.72 μs 26.22 μs 0.76 0.08 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Reader.Tests.Perf_Base64*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Reader.Tests.Perf_Base64*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Reader.Tests.Perf_Base64*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Reader.Tests.Perf_Base64*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Reader.Tests.Perf_Base64.ReadBase64EncodedByteArray_NoEscaping(NumberOfBytes: 100)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 48.481268082086444 (T) = (0 -3683.990299731998) / Math.Sqrt((16130.880527788524 / (109)) + (991.0585753978896 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.18889483125723758 = (4541.939124173373 - 3683.990299731998) / 4541.939124173373 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Reader.Tests.Perf_Base64.ReadBase64EncodedByteArray_HeavyEscaping(NumberOfBytes: 100)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 61.057473690445555 (T) = (0 -3676.201479552788) / Math.Sqrt((15601.576457769852 / (108)) + (341.2551186761707 / (6))) is greater than 1.9813718148759247 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (108) + (6) - 2, .975) and 0.19072057599227313 = (4542.561407711866 - 3676.201479552788) / 4542.561407711866 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Reader.Tests.Perf_Base64.ReadBase64EncodedByteArray_NoEscaping(NumberOfBytes: 1000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 58.56129040399435 (T) = (0 -26490.772701873986) / Math.Sqrt((1886237.6501402762 / (109)) + (54884.154734159725 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.26445729690342296 = (36015.27496683729 - 26490.772701873986) / 36015.27496683729 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Reader.Tests.Perf_Base64.ReadBase64EncodedByteArray_HeavyEscaping(NumberOfBytes: 1000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 57.3184602005205 (T) = (0 -26467.285076469412) / Math.Sqrt((1871425.2667720492 / (109)) + (61569.97479684638 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.26398991807363154 = (35960.49256172722 - 26467.285076469412) / 35960.49256172722 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 74.33 secs 60.22 secs 0.81 0.15 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase, NonBacktracking)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 14.316412298371835 (T) = (0 -59055200353.794304) / Math.Sqrt((5.314629341242015E+18 / (109)) + (5.61642531636431E+18 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.1939243182012251 = (73262600134.5325 - 59055200353.794304) / 73262600134.5325 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromReader - Duration of single invocation 869.17 μs 720.64 μs 0.83 0.02 False
DeserializeFromUtf8Bytes - Duration of single invocation 596.17 μs 491.76 μs 0.82 0.05 False
DeserializeFromStream - Duration of single invocation 644.19 μs 524.15 μs 0.81 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 601.73 μs 495.69 μs 0.82 0.02 False
DeserializeFromReader - Duration of single invocation 878.87 μs 746.47 μs 0.85 0.06 False
DeserializeFromStream - Duration of single invocation 624.56 μs 519.16 μs 0.83 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>.DeserializeFromReader(Mode: SourceGen)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 42.134436780100984 (T) = (0 -724244.2791895604) / Math.Sqrt((834454167.8676476 / (89)) + (24850163.410798818 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.1762058599253595 = (879156.8717931639 - 724244.2791895604) / 879156.8717931639 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 18.86504327651914 (T) = (0 -499412.04991802707) / Math.Sqrt((988184626.167459 / (89)) + (87287241.08285023 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.16059223037345502 = (594957.621300333 - 499412.04991802707) / 594957.621300333 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 31.97007325614024 (T) = (0 -528371.4418325807) / Math.Sqrt((788322715.8230438 / (89)) + (9039945.237300728 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.16303467804798505 = (631294.305719005 - 528371.4418325807) / 631294.305719005 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 28.504253369605543 (T) = (0 -498533.3087631723) / Math.Sqrt((1004131078.71324 / (89)) + (8232639.787104474 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.1690167938282721 = (599931.8699349831 - 498533.3087631723) / 599931.8699349831 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 26.003148023610482 (T) = (0 -725966.5257905786) / Math.Sqrt((789226692.2877406 / (89)) + (144420960.90435976 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.17051600099270994 = (875202.5676919637 - 725966.5257905786) / 875202.5676919637 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 31.49616483014167 (T) = (0 -524734.3182622005) / Math.Sqrt((766654577.8612329 / (89)) + (9263443.249048218 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.16058318011136866 = (625117.7077102398 - 524734.3182622005) / 625117.7077102398 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.ReadJson<ImmutableDictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromStream - Duration of single invocation 662.41 μs 611.43 μs 0.92 0.02 False
DeserializeFromReader - Duration of single invocation 790.44 μs 706.12 μs 0.89 0.03 False
DeserializeFromString - Duration of single invocation 694.59 μs 617.27 μs 0.89 0.02 False
DeserializeFromString - Duration of single invocation 694.34 μs 614.90 μs 0.89 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 639.77 μs 587.53 μs 0.92 0.02 False
DeserializeFromStream - Duration of single invocation 662.32 μs 608.78 μs 0.92 0.02 False
DeserializeFromUtf8Bytes - Duration of single invocation 642.77 μs 592.51 μs 0.92 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<ImmutableDictionary<String, String>>.DeserializeFromStream(Mode: SourceGen)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 16.62231072872593 (T) = (0 -616410.6295054945) / Math.Sqrt((361441599.0179486 / (89)) + (17431829.02162954 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.06644564714486012 = (660283.6006498095 - 616410.6295054945) / 660283.6006498095 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 12.31202619859616 (T) = (0 -717503.5190902847) / Math.Sqrt((405959856.0908404 / (89)) + (104918258.50292596 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.07456472462919114 = (775314.6418616809 - 717503.5190902847) / 775314.6418616809 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 33.64384355302179 (T) = (0 -620815.6644307082) / Math.Sqrt((276891062.3436528 / (89)) + (9671360.047307137 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.10536559726583483 = (693932.2504627396 - 620815.6644307082) / 693932.2504627396 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 17.508300155810904 (T) = (0 -623226.6108142552) / Math.Sqrt((315785094.20529085 / (89)) + (76441072.06083438 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.10183400553000437 = (693888.0058379619 - 623226.6108142552) / 693888.0058379619 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 10.374963180281686 (T) = (0 -597710.6464875344) / Math.Sqrt((456890997.5971396 / (89)) + (49805112.27945329 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.059816148692205855 = (635738.0481021025 - 597710.6464875344) / 635738.0481021025 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 15.601725722052185 (T) = (0 -615257.509018679) / Math.Sqrt((331262817.7999987 / (89)) + (26895359.239686467 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.06771625111347146 = (659946.6200644501 - 615257.509018679) / 659946.6200644501 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 8.288434070285088 (T) = (0 -598886.3951942156) / Math.Sqrt((440741520.76003367 / (89)) + (89282664.26495636 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.058055470650726575 = (635797.9440763313 - 598886.3951942156) / 635797.9440763313 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.IO.Tests.Perf_File

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AppendAllLines - Duration of single invocation 45.02 μs 41.22 μs 0.92 0.03 False
AppendAllText - Duration of single invocation 119.31 μs 77.38 μs 0.65 0.32 False
ReadAllLines - Duration of single invocation 175.16 μs 128.20 μs 0.73 0.05 False
ReadAllLinesAsync - Duration of single invocation 426.57 μs 353.27 μs 0.83 0.07 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_File*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.Perf_File*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_File*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.Perf_File*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_File.AppendAllLines

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 33.76457367866698 (T) = (0 -40744.24383849207) / Math.Sqrt((3056004.4376065144 / (109)) + (127343.53112065271 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.15535321760942492 = (48238.204049241765 - 40744.24383849207) / 48238.204049241765 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_File.AppendAllText(size: 10000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 40.9868578232707 (T) = (0 -73550.43154166666) / Math.Sqrt((106535138.05701566 / (108)) + (4353408.880705925 / (6))) is greater than 1.9813718148759247 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (108) + (6) - 2, .975) and 0.4216783567178152 = (127179.10940396649 - 73550.43154166666) / 127179.10940396649 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_File.ReadAllLines

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 23.633768689124917 (T) = (0 -128491.76169537146) / Math.Sqrt((252372574.57318002 / (109)) + (72018.83561988953 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.21911630751483144 = (164546.60653297216 - 128491.76169537146) / 164546.60653297216 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_File.ReadAllLinesAsync

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 13.454123868325212 (T) = (0 -358623.4499142198) / Math.Sqrt((773435903.8126501 / (109)) + (50405571.38423705 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.1286806639948601 = (411586.6997264758 - 358623.4499142198) / 411586.6997264758 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ExclusiveOrOperatorBenchmark - Duration of single invocation 62.61 ns 11.22 ns 0.18 0.40 False
EqualsAllBenchmark - Duration of single invocation 56.72 ns 9.20 ns 0.16 0.34 False
NegateBenchmark - Duration of single invocation 261.80 ns 7.78 ns 0.03 0.31 False
BitwiseOrOperatorBenchmark - Duration of single invocation 62.38 ns 11.22 ns 0.18 0.34 False
AndNotBenchmark - Duration of single invocation 67.54 ns 12.28 ns 0.18 0.38 False
InequalityOperatorBenchmark - Duration of single invocation 49.22 ns 8.87 ns 0.18 0.38 False
MultiplyOperatorBenchmark - Duration of single invocation 256.85 ns 12.23 ns 0.05 0.26 False
SubtractBenchmark - Duration of single invocation 259.31 ns 11.16 ns 0.04 0.23 False
BitwiseOrBenchmark - Duration of single invocation 65.24 ns 12.64 ns 0.19 0.42 False
ConditionalSelectBenchmark - Duration of single invocation 91.16 ns 14.71 ns 0.16 0.23 False
XorBenchmark - Duration of single invocation 62.44 ns 11.22 ns 0.18 0.30 False
SubtractionOperatorBenchmark - Duration of single invocation 256.72 ns 11.25 ns 0.04 0.18 False
OnesComplementBenchmark - Duration of single invocation 43.70 ns 7.50 ns 0.17 0.45 False
AddBenchmark - Duration of single invocation 257.10 ns 11.15 ns 0.04 0.24 False
BitwiseAndBenchmark - Duration of single invocation 63.51 ns 11.28 ns 0.18 0.34 False
EqualityOperatorBenchmark - Duration of single invocation 48.31 ns 9.30 ns 0.19 0.30 False
UnaryNegateOperatorBenchmark - Duration of single invocation 257.94 ns 7.66 ns 0.03 0.27 False
OnesComplementOperatorBenchmark - Duration of single invocation 44.10 ns 7.39 ns 0.17 0.33 False
MultiplyBenchmark - Duration of single invocation 273.10 ns 10.88 ns 0.04 0.19 False
EqualsStaticBenchmark - Duration of single invocation 302.14 ns 11.09 ns 0.04 0.23 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 40.522892001760816 (T) = (0 -11.062948382832952) / Math.Sqrt((197.4871692351833 / (99)) + (0.03437847836407814 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8382109521208276 = (68.3788459593075 - 11.062948382832952) / 68.3788459593075 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 103.26246934035812 (T) = (0 -9.342414169734019) / Math.Sqrt((19.250690892966468 / (99)) + (0.020703188155788234 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8309980960218616 = (55.279934425724164 - 9.342414169734019) / 55.279934425724164 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 78.6552883802721 (T) = (0 -7.484461823233617) / Math.Sqrt((1112.3695671138612 / (99)) + (0.05356157373786582 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9724068516846436 = (271.243489060953 - 7.484461823233617) / 271.243489060953 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 41.65804088102057 (T) = (0 -10.962167852897307) / Math.Sqrt((182.80364017395164 / (99)) + (0.1878070244062769 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8389039540965311 = (68.04740483491443 - 10.962167852897307) / 68.04740483491443 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 43.16798787628582 (T) = (0 -11.366006840490334) / Math.Sqrt((177.09317781405096 / (99)) + (0.2155660012886278 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8368797870440721 = (69.67871506862993 - 11.366006840490334) / 69.67871506862993 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 66.0673538607568 (T) = (0 -9.442034587720752) / Math.Sqrt((28.973427105142022 / (99)) + (0.6187540304661884 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8148840031831391 = (51.00604350828714 - 9.442034587720752) / 51.00604350828714 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.MultiplyOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 79.87900606215554 (T) = (0 -11.654823327411394) / Math.Sqrt((1084.6109575374269 / (99)) + (0.4779443249188011 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9579261240120196 = (277.0085487427149 - 11.654823327411394) / 277.0085487427149 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 79.19674199781932 (T) = (0 -11.291947839740752) / Math.Sqrt((1125.7263391916404 / (98)) + (1.0207974622481972 / (6))) is greater than 1.9834952585625159 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (98) + (6) - 2, .975) and 0.9599135381617236 = (281.689810522482 - 11.291947839740752) / 281.689810522482 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 38.67699361046143 (T) = (0 -11.700033271005049) / Math.Sqrt((195.5139416704893 / (99)) + (0.9412466826962419 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8283712723001121 = (68.17059957155816 - 11.700033271005049) / 68.17059957155816 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 55.98204369933196 (T) = (0 -14.713698185898997) / Math.Sqrt((205.02101022512775 / (99)) + (0.038008341060980125 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8457664833028983 = (95.39883743165328 - 14.713698185898997) / 95.39883743165328 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 42.45358795775257 (T) = (0 -11.021079316597351) / Math.Sqrt((173.36002914142324 / (99)) + (0.04124062851628285 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8362635620387651 = (67.30987588240207 - 11.021079316597351) / 67.30987588240207 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 81.58166718413594 (T) = (0 -11.120025568605483) / Math.Sqrt((1060.1553291862917 / (99)) + (0.04121345384323337 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9600249217692246 = (278.17395389221605 - 11.120025568605483) / 278.17395389221605 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 33.13813947890258 (T) = (0 -7.506629579554953) / Math.Sqrt((170.4318423632064 / (99)) + (0.020853656581258494 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8528981000986315 = (51.03013342851541 - 7.506629579554953) / 51.03013342851541 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 80.353513464474 (T) = (0 -11.026441221269977) / Math.Sqrt((1122.8778919639335 / (99)) + (0.022072211379593178 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9608556091189995 = (281.6863661205129 - 11.026441221269977) / 281.6863661205129 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 41.92687714465987 (T) = (0 -11.44924926105387) / Math.Sqrt((169.72403200921383 / (99)) + (0.6322298309021821 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8316482560503804 = (68.0079041205545 - 11.44924926105387) / 68.0079041205545 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 90.24533897504432 (T) = (0 -9.301800790428194) / Math.Sqrt((19.8062823921044 / (90)) + (0.011857561339231373 / (6))) is greater than 1.98552344186624 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (90) + (6) - 2, .975) and 0.8205220942159935 = (51.826996475112054 - 9.301800790428194) / 51.826996475112054 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 75.09150079645634 (T) = (0 -7.626059911022185) / Math.Sqrt((1186.9414307214988 / (98)) + (0.02214006150234935 / (6))) is greater than 1.9834952585625159 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (98) + (6) - 2, .975) and 0.9716500972934495 = (268.9977454229552 - 7.626059911022185) / 268.9977454229552 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 32.63529863235763 (T) = (0 -7.5399135512444815) / Math.Sqrt((176.91893892174653 / (99)) + (0.03412840159245337 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8528408878688505 = (51.23647079716576 - 7.5399135512444815) / 51.23647079716576 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.MultiplyBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 82.24952314258235 (T) = (0 -11.214980111230117) / Math.Sqrt((1074.011895326489 / (99)) + (0.0791800305046001 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9602709679973452 = (282.2867697979823 - 11.214980111230117) / 282.2867697979823 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.EqualsStaticBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 88.70734680433803 (T) = (0 -11.09621973199526) / Math.Sqrt((1106.0489743060327 / (99)) + (0.03581391231751657 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9639356639437812 = (307.6784697962522 - 11.09621973199526) / 307.6784697962522 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AndNotBenchmark - Duration of single invocation 62.74 ns 10.98 ns 0.18 0.37 False
EqualityOperatorBenchmark - Duration of single invocation 50.19 ns 9.10 ns 0.18 0.31 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 79.28 ns 10.92 ns 0.14 0.30 False
InequalityOperatorBenchmark - Duration of single invocation 50.10 ns 9.13 ns 0.18 0.43 False
BitwiseAndBenchmark - Duration of single invocation 62.27 ns 11.08 ns 0.18 0.38 False
XorBenchmark - Duration of single invocation 61.84 ns 12.14 ns 0.20 0.40 False
BitwiseOrOperatorBenchmark - Duration of single invocation 81.93 ns 11.01 ns 0.13 0.44 False
EqualsBenchmark - Duration of single invocation 61.59 ns 10.50 ns 0.17 0.29 False
BitwiseAndOperatorBenchmark - Duration of single invocation 63.73 ns 12.51 ns 0.20 0.36 False
OnesComplementOperatorBenchmark - Duration of single invocation 43.79 ns 7.88 ns 0.18 0.52 False
OnesComplementBenchmark - Duration of single invocation 47.52 ns 7.64 ns 0.16 0.37 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.AndNotBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 44.6505996645998 (T) = (0 -11.171921184724596) / Math.Sqrt((170.97256571705543 / (99)) + (0.011836131166915516 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8401341534189907 = (69.8831015107621 - 11.171921184724596) / 69.8831015107621 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 102.46721257503442 (T) = (0 -9.172815985825263) / Math.Sqrt((16.73883183600428 / (98)) + (0.00397819820358136 / (6))) is greater than 1.9834952585625159 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (98) + (6) - 2, .975) and 0.8222428652013236 = (51.603081902811866 - 9.172815985825263) / 51.603081902811866 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 42.391180303472815 (T) = (0 -11.102008874422703) / Math.Sqrt((176.84818929273922 / (99)) + (0.010809840601552207 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8362250022663806 = (67.78817907529549 - 11.102008874422703) / 67.78817907529549 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 72.47646302236251 (T) = (0 -9.240526301338821) / Math.Sqrt((33.025890187802275 / (99)) + (0.00933929939814801 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8195167059958973 = (51.198790183476845 - 9.240526301338821) / 51.198790183476845 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 39.28435980683021 (T) = (0 -11.71387386089487) / Math.Sqrt((186.0661287389036 / (99)) + (1.0149165480410676 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8275886367795541 = (67.94142591354294 - 11.71387386089487) / 67.94142591354294 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 38.19743997671632 (T) = (0 -11.716394012331328) / Math.Sqrt((196.80170984222886 / (99)) + (0.901641180060032 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8266049687118082 = (67.5705291281274 - 11.716394012331328) / 67.5705291281274 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/3/2022 11:38:23 AM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 40.93913198041102 (T) = (0 -11.098913698527445) / Math.Sqrt((195.23690701143911 / (99)) + (0.016774098687252346 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8382811740271036 = (68.63093169120327 - 11.098913698527445) / 68.63093169120327 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 90.18539537178145 (T) = (0 -10.578323770922804) / Math.Sqrt((32.5748965225139 / (90)) + (0.012925135272666649 / (6))) is greater than 1.98552344186624 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (90) + (6) - 2, .975) and 0.8372481294176568 = (64.996634036048 - 10.578323770922804) / 64.996634036048 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 40.770138715899385 (T) = (0 -11.756628845030546) / Math.Sqrt((174.98519715975664 / (99)) + (0.8428473920612566 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8272927904681046 = (68.07260030948126 - 11.756628845030546) / 68.07260030948126 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 32.01204788665543 (T) = (0 -7.6051885368884795) / Math.Sqrt((184.40267779384465 / (99)) + (0.07277550359651082 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8521454180791275 = (51.43694864294807 - 7.6051885368884795) / 51.43694864294807 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 32.76547136588259 (T) = (0 -7.512983991047573) / Math.Sqrt((173.2897110780689 / (99)) + (0.027238429558674136 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8524515923956434 = (50.9187737979067 - 7.512983991047573) / 50.9187737979067 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOf_Word_NotFound - Duration of single invocation 2.14 μs 1.60 μs 0.75 0.24 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.StringSearch*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Globalization.Tests.StringSearch*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.StringSearch*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Globalization.Tests.StringSearch*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, OrdinalIgnoreCase, False))

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 31.9964084575012 (T) = (0 -1565.4511641342926) / Math.Sqrt((8681.993526679518 / (109)) + (1428.8856998631684 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.2670588755637819 = (2135.848449407782 - 1565.4511641342926) / 2135.848449407782 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Load - Duration of single invocation 1.51 ms 1.30 ms 0.86 0.03 False
Load - Duration of single invocation 1.95 ms 1.81 ms 0.93 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "deep.xml")

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 1/3/2023 12:19:11 AM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 21.17323395219339 (T) = (0 -1316798.5200636617) / Math.Sqrt((5782879157.471118 / (109)) + (128193030.35042533 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.12181409285341879 = (1499453.0307850523 - 1316798.5200636617) / 1499453.0307850523 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "repeated.xml")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 12.150050728942656 (T) = (0 -1834876.4965143837) / Math.Sqrt((2700983576.836433 / (98)) + (231497574.95166034 / (6))) is greater than 1.9834952585625159 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (98) + (6) - 2, .975) and 0.051101784383482396 = (1933691.5870605034 - 1834876.4965143837) / 1933691.5870605034 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.IO.Tests.Perf_Directory

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetCurrentDirectory - Duration of single invocation 4.13 μs 2.78 μs 0.67 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_Directory*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.Perf_Directory*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_Directory*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.Perf_Directory*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_Directory.GetCurrentDirectory

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 12.83794374854484 (T) = (0 -2783.196693220101) / Math.Sqrt((486606.6877551157 / (109)) + (621.2944819323452 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.23765965766028252 = (3650.8584665454327 - 2783.196693220101) / 3650.8584665454327 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.WriteJson<ImmutableDictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeObjectProperty - Duration of single invocation 558.04 μs 513.54 μs 0.92 0.02 False
SerializeToString - Duration of single invocation 509.19 μs 462.95 μs 0.91 0.03 False
SerializeToString - Duration of single invocation 557.62 μs 522.55 μs 0.94 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<ImmutableDictionary<String, String>>.SerializeObjectProperty(Mode: Reflection)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 16.72632901813689 (T) = (0 -517114.896492889) / Math.Sqrt((298973263.55168027 / (89)) + (37236570.17141905 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.09094028722502426 = (568845.9066284605 - 517114.896492889) / 568845.9066284605 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.SerializeToString(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 18.768626162904436 (T) = (0 -464835.7082546845) / Math.Sqrt((222873222.4409983 / (89)) + (31324795.753339835 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.10089976488238725 = (517000.98620692565 - 464835.7082546845) / 517000.98620692565 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.SerializeToString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 17.551780813671765 (T) = (0 -509950.49038849253) / Math.Sqrt((228769809.01961553 / (89)) + (39943853.84663211 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.09465732409933185 = (563267.9249116088 - 509950.49038849253) / 563267.9249116088 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in MicroBenchmarks.Serializers.Json_ToString<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 742.49 μs 608.03 μs 0.82 0.03 False
SystemTextJson_Reflection_ - Duration of single invocation 818.14 μs 692.94 μs 0.85 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_ToString<IndexViewModel>.SystemTextJson_SourceGen_

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 31.582414279513696 (T) = (0 -608207.4347619048) / Math.Sqrt((522503874.5049002 / (89)) + (61905650.5982646 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.17282069304819797 = (735278.8321109968 - 608207.4347619048) / 735278.8321109968 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_ToString&lt;IndexViewModel&gt;.SystemTextJson_Reflection_

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 48.650820310562445 (T) = (0 -689589.8856725729) / Math.Sqrt((560629817.5773371 / (89)) + (6321848.170872868 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.1605846802039649 = (821512.1518631951 - 689589.8856725729) / 821512.1518631951 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArrayAnd - Duration of single invocation 1.54 μs 1.06 μs 0.69 0.14 False
BitArrayOr - Duration of single invocation 1.55 μs 971.74 ns 0.63 0.09 False
BitArrayNot - Duration of single invocation 1.12 μs 699.07 ns 0.63 0.05 False
BitArrayXor - Duration of single invocation 1.54 μs 1.01 μs 0.66 0.12 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Collections.Tests.Perf_BitArray*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Collections.Tests.Perf_BitArray*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 17.21678390208405 (T) = (0 -1061.2873056027381) / Math.Sqrt((940.745775694993 / (109)) + (4780.119506408027 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.31523981719440297 = (1549.8671392580618 - 1061.2873056027381) / 1549.8671392580618 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 41.46052338036708 (T) = (0 -1002.3402851035971) / Math.Sqrt((977.0708434248028 / (109)) + (986.4841878118665 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.3526036562987698 = (1548.263741146755 - 1002.3402851035971) / 1548.263741146755 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 27.433677118065177 (T) = (0 -722.4239511775604) / Math.Sqrt((219.48443610982744 / (109)) + (1245.177932598945 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.35471540332330576 = (1119.5431518095188 - 722.4239511775604) / 1119.5431518095188 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayXor(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 17.335316831355094 (T) = (0 -1071.2601088149356) / Math.Sqrt((1607.2695202610228 / (108)) + (4474.471695634983 / (6))) is greater than 1.9813718148759247 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (108) + (6) - 2, .975) and 0.3085785255080338 = (1549.3590354596122 - 1071.2601088149356) / 1549.3590354596122 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.IO.Tests.StreamReaderReadToEndTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadToEnd - Duration of single invocation 639.01 ms 287.89 ms 0.45 0.11 False
ReadToEndAsync - Duration of single invocation 672.84 ms 334.80 ms 0.50 0.12 False
ReadToEnd - Duration of single invocation 609.44 ms 267.25 ms 0.44 0.10 False
ReadToEnd - Duration of single invocation 624.19 ms 268.52 ms 0.43 0.14 False
ReadToEndAsync - Duration of single invocation 681.69 ms 338.83 ms 0.50 0.16 False
ReadToEnd - Duration of single invocation 623.30 ms 284.34 ms 0.46 0.17 False
ReadToEnd - Duration of single invocation 623.85 ms 269.52 ms 0.43 0.09 False
ReadToEnd - Duration of single invocation 625.21 ms 265.37 ms 0.42 0.05 False
ReadToEnd - Duration of single invocation 624.67 ms 270.39 ms 0.43 0.13 False
ReadToEnd - Duration of single invocation 636.48 ms 266.65 ms 0.42 0.09 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StreamReaderReadToEndTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StreamReaderReadToEndTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StreamReaderReadToEndTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StreamReaderReadToEndTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 1, 1])

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 11.640292585176203 (T) = (0 -275443840.8225885) / Math.Sqrt((32568916113348816 / (109)) + (82679980550224.2 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.4276403195986466 = (481242565.2160547 - 275443840.8225885) / 481242565.2160547 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [   1,    1])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 12.126432828021652 (T) = (0 -320837993.67777777) / Math.Sqrt((31416080775191908 / (108)) + (50084489088772.33 / (6))) is greater than 1.9813718148759247 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (108) + (6) - 2, .975) and 0.395337489445933 = (530606723.71400386 - 320837993.67777777) / 530606723.71400386 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [  33,  128])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 12.123526503497686 (T) = (0 -272453902.9595238) / Math.Sqrt((31895358749138756 / (109)) + (34269609658594.074 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.4345718048775625 = (481854115.7122997 - 272453902.9595238) / 481854115.7122997 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [   9,   32])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 12.05169776091731 (T) = (0 -274070703.08174604) / Math.Sqrt((32108025745955452 / (100)) + (91481575343806.47 / (6))) is greater than 1.9830375264834914 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (100) + (6) - 2, .975) and 0.44642190810132376 = (495089504.24994487 - 274070703.08174604) / 495089504.24994487 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [   0,    0])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 11.826997136094452 (T) = (0 -324839695.2394994) / Math.Sqrt((30732517897550488 / (109)) + (86993856164637.89 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.3853241937575106 = (528473208.05619967 - 324839695.2394994) / 528473208.05619967 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [   0,    0])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 11.835735829588085 (T) = (0 -273371372.03492063) / Math.Sqrt((32426132743226268 / (109)) + (78428253786567.62 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.43277916687812623 = (481948750.94826376 - 273371372.03492063) / 481948750.94826376 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 12.299760485125734 (T) = (0 -269642739.46068376) / Math.Sqrt((32227110786439364 / (109)) + (2911220032991.087 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.4397712399576004 = (481308277.426308 - 269642739.46068376) / 481308277.426308 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 12.320075768780441 (T) = (0 -268272083.2555556) / Math.Sqrt((32264751778530820 / (109)) + (6565849491621.785 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.44183079140541154 = (480628596.3552819 - 268272083.2555556) / 480628596.3552819 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [   1,    8])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 12.559691328722597 (T) = (0 -268755975.42539686) / Math.Sqrt((32460876783508056 / (102)) + (1788785886170.3699 / (6))) is greater than 1.9825972617651593 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (102) + (6) - 2, .975) and 0.4547655118391062 = (492918150.37586063 - 268755975.42539686) / 492918150.37586063 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 12.005576913244866 (T) = (0 -270506807.68333334) / Math.Sqrt((32556092306805588 / (109)) + (62708053192663.44 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.43830517545220926 = (481590350.954565 - 270506807.68333334) / 481590350.954565 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeObjectProperty - Duration of single invocation 14.56 μs 13.57 μs 0.93 0.04 False
SerializeToString - Duration of single invocation 12.06 μs 11.04 μs 0.92 0.04 False
SerializeToString - Duration of single invocation 8.82 μs 8.28 μs 0.94 0.08 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ClassRecord&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ClassRecord&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ClassRecord&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ClassRecord&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>.SerializeObjectProperty(Mode: Reflection)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 15.683199593408892 (T) = (0 -13621.835093601816) / Math.Sqrt((69053.27939685101 / (16)) + (1227.4992928286913 / (6))) is greater than 2.08596344725343 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (6) - 2, .975) and 0.07184685433899524 = (14676.279617520151 - 13621.835093601816) / 14676.279617520151 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;ClassRecord&gt;.SerializeToString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 10.151543980580195 (T) = (0 -11149.704653716835) / Math.Sqrt((65510.30509410932 / (16)) + (29845.191158145542 / (6))) is greater than 2.08596344725343 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (6) - 2, .975) and 0.07978613973124901 = (12116.427642657472 - 11149.704653716835) / 12116.427642657472 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;ClassRecord&gt;.SerializeToString(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 12.01881877387964 (T) = (0 -8058.632024994302) / Math.Sqrt((75013.8850266778 / (16)) + (12548.741185989918 / (6))) is greater than 2.08596344725343 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (6) - 2, .975) and 0.10937197394605339 = (9048.257846431366 - 8058.632024994302) / 9048.257846431366 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 473.15 μs 427.53 μs 0.90 0.03 False
SerializeToString - Duration of single invocation 450.64 μs 403.78 μs 0.90 0.03 False
SerializeObjectProperty - Duration of single invocation 475.55 μs 444.33 μs 0.93 0.03 False
SerializeObjectProperty - Duration of single invocation 453.60 μs 409.69 μs 0.90 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;Dictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;Dictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;Dictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;Dictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeToString(Mode: Reflection)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 23.582756753265585 (T) = (0 -426947.6526849996) / Math.Sqrt((201286459.04535925 / (89)) + (15673692.847592706 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.10868970444423559 = (479011.2431258097 - 426947.6526849996) / 479011.2431258097 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Dictionary&lt;String, String&gt;&gt;.SerializeToString(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 31.791122331995986 (T) = (0 -401992.4247354497) / Math.Sqrt((199273755.4649417 / (89)) + (3278848.460910908 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.11659972999232701 = (455051.28126342787 - 401992.4247354497) / 455051.28126342787 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Dictionary&lt;String, String&gt;&gt;.SerializeObjectProperty(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 17.63036933158551 (T) = (0 -433024.9352069012) / Math.Sqrt((257350694.9346652 / (89)) + (32510104.5472807 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.10503916477607107 = (483847.91620356624 - 433024.9352069012) / 483847.91620356624 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Dictionary&lt;String, String&gt;&gt;.SerializeObjectProperty(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 29.95330809725104 (T) = (0 -407778.68789925845) / Math.Sqrt((254696154.74570954 / (89)) + (1751722.896811638 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.11539345640547714 = (460971.819451261 - 407778.68789925845) / 460971.819451261 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AndNotBenchmark - Duration of single invocation 61.63 ns 12.81 ns 0.21 0.39 False
EqualsBenchmark - Duration of single invocation 59.19 ns 11.42 ns 0.19 0.43 False
BitwiseOrOperatorBenchmark - Duration of single invocation 77.83 ns 11.92 ns 0.15 0.32 False
XorBenchmark - Duration of single invocation 61.72 ns 10.99 ns 0.18 0.36 False
InequalityOperatorBenchmark - Duration of single invocation 47.14 ns 10.41 ns 0.22 0.42 False
BitwiseAndBenchmark - Duration of single invocation 63.83 ns 11.41 ns 0.18 0.46 False
EqualityOperatorBenchmark - Duration of single invocation 47.71 ns 9.34 ns 0.20 0.41 False
OnesComplementOperatorBenchmark - Duration of single invocation 47.89 ns 7.15 ns 0.15 0.49 False
OnesComplementBenchmark - Duration of single invocation 47.76 ns 8.97 ns 0.19 0.60 False
BitwiseAndOperatorBenchmark - Duration of single invocation 65.37 ns 11.13 ns 0.17 0.42 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 64.62 ns 10.93 ns 0.17 0.38 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.AndNotBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 43.11344151123233 (T) = (0 -11.459276165289358) / Math.Sqrt((172.95086743683814 / (99)) + (0.6533168256017138 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8367456336538144 = (70.19276985823234 - 11.459276165289358) / 70.19276985823234 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 92.07596175177379 (T) = (0 -10.703165425672571) / Math.Sqrt((25.779552260829718 / (90)) + (0.4138661224350095 / (6))) is greater than 1.98552344186624 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (90) + (6) - 2, .975) and 0.8368319784172323 = (65.5959747617786 - 10.703165425672571) / 65.5959747617786 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 41.66953384073871 (T) = (0 -11.495622415471578) / Math.Sqrt((179.78572023077612 / (99)) + (0.09918848351195568 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.830708637075301 = (67.90436450431817 - 11.495622415471578) / 67.90436450431817 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 39.82130926034886 (T) = (0 -11.517807756185206) / Math.Sqrt((179.91885302300796 / (99)) + (1.0309114332689864 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8298224231163127 = (67.68111267712656 - 11.517807756185206) / 67.68111267712656 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 64.6651956201301 (T) = (0 -9.931033820625208) / Math.Sqrt((33.34069045129278 / (99)) + (0.44249299029433187 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8066514994819611 = (51.36338680681246 - 9.931033820625208) / 51.36338680681246 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 40.64653179124549 (T) = (0 -11.730566686410834) / Math.Sqrt((189.40340018274065 / (99)) + (0.6824734976863136 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8314546747006485 = (69.59888484344671 - 11.730566686410834) / 69.59888484344671 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 59.94876119286645 (T) = (0 -10.014970201122788) / Math.Sqrt((23.087152431668827 / (98)) + (1.4895065422971983 / (6))) is greater than 1.9834952585625159 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (98) + (6) - 2, .975) and 0.8063402127741125 = (51.71424767415024 - 10.014970201122788) / 51.71424767415024 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 33.378415390854904 (T) = (0 -7.6855873376626525) / Math.Sqrt((164.89570120229652 / (99)) + (0.3395228469967345 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8507332942038243 = (51.488959287125645 - 7.6855873376626525) / 51.488959287125645 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 31.743746572006696 (T) = (0 -7.972074737413696) / Math.Sqrt((181.0472662121846 / (99)) + (0.5275767189254472 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8464533637193674 = (51.919566136527905 - 7.972074737413696) / 51.919566136527905 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 42.851998393143205 (T) = (0 -11.305945189447172) / Math.Sqrt((173.42393672096975 / (99)) + (0.11360232614058977 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8345340907572493 = (68.32794284447147 - 11.305945189447172) / 68.32794284447147 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 41.95862828155149 (T) = (0 -11.290418486182917) / Math.Sqrt((172.7360781710364 / (99)) + (0.32520814699410094 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8329037322000795 = (67.56834628827232 - 11.290418486182917) / 67.56834628827232 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ExclusiveOrOperatorBenchmark - Duration of single invocation 60.73 ns 12.20 ns 0.20 0.37 False
BitwiseOrOperatorBenchmark - Duration of single invocation 65.09 ns 12.87 ns 0.20 0.41 False
OnesComplementBenchmark - Duration of single invocation 46.14 ns 7.80 ns 0.17 0.43 False
OnesComplementOperatorBenchmark - Duration of single invocation 43.67 ns 7.33 ns 0.17 0.49 False
BitwiseAndOperatorBenchmark - Duration of single invocation 62.10 ns 10.89 ns 0.18 0.39 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 41.69462577340173 (T) = (0 -11.481982308366469) / Math.Sqrt((172.7225214779986 / (99)) + (0.31002367253020424 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.829553979465593 = (67.36433195897735 - 11.481982308366469) / 67.36433195897735 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 39.73401559310126 (T) = (0 -11.913851608791447) / Math.Sqrt((177.97858767776637 / (99)) + (0.8870584284434597 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8230708349399455 = (67.33684412486524 - 11.913851608791447) / 67.33684412486524 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 33.07412573169258 (T) = (0 -7.931342715403789) / Math.Sqrt((159.21765472637043 / (99)) + (0.43635807179183345 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8439107843066551 = (50.8128808269869 - 7.931342715403789) / 50.8128808269869 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 32.469903670685156 (T) = (0 -7.517610835404252) / Math.Sqrt((179.70545265081284 / (99)) + (0.05460179998232874 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8536681197049195 = (51.373704897694736 - 7.517610835404252) / 51.373704897694736 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 42.441408097024855 (T) = (0 -11.82785619676262) / Math.Sqrt((171.1595578754425 / (99)) + (0.40510785066180766 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8278634326336033 = (68.71204868159562 - 11.82785619676262) / 68.71204868159562 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in MicroBenchmarks.Serializers.Xml_FromStream<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 11.22 ms 10.36 ms 0.92 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;CollectionsOfPrimitives&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;CollectionsOfPrimitives&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;CollectionsOfPrimitives&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;CollectionsOfPrimitives&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<CollectionsOfPrimitives>.XmlSerializer_

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 9.700906087762492 (T) = (0 -10539852.002803659) / Math.Sqrt((93200470312.96794 / (109)) + (18636646688.13075 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.05475612908539181 = (11150405.019400345 - 10539852.002803659) / 11150405.019400345 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in BenchmarksGame.ReverseComplement_6

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
RunBench - Duration of single invocation 14.28 ms 10.17 ms 0.71 0.10 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'BenchmarksGame.ReverseComplement_6*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'BenchmarksGame.ReverseComplement_6*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'BenchmarksGame.ReverseComplement_6*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'BenchmarksGame.ReverseComplement_6*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

BenchmarksGame.ReverseComplement_6.RunBench

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 35.884130739993765 (T) = (0 -10257409.395528955) / Math.Sqrt((363155373128.11755 / (101)) + (52391678205.73126 / (6))) is greater than 1.9828152737947098 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (101) + (6) - 2, .975) and 0.2797575138604117 = (14241605.560520897 - 10257409.395528955) / 14241605.560520897 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in MicroBenchmarks.Serializers.Json_ToStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
JsonNet_ - Duration of single invocation 700.19 μs 622.86 μs 0.89 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToStream&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_ToStream&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToStream&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_ToStream&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_ToStream<IndexViewModel>.JsonNet_

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 12.787361942348454 (T) = (0 -638959.5439020704) / Math.Sqrt((724251469.5442733 / (107)) + (209783977.70125455 / (6))) is greater than 1.9815667570746824 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (107) + (6) - 2, .975) and 0.11448340807512308 = (721566.9923396271 - 638959.5439020704) / 721566.9923396271 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Net.Primitives.Tests.IPAddressPerformanceTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryWriteBytes - Duration of single invocation 569.55 ns 78.54 ns 0.14 0.38 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Net.Primitives.Tests.IPAddressPerformanceTests.TryWriteBytes(address: 1020:3040:5060:7080:9010:1112:1314:1516)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 149.5792711324893 (T) = (0 -80.6421422171142) / Math.Sqrt((772.4122055391231 / (100)) + (8.403172702732027 / (6))) is greater than 1.9830375264834914 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (100) + (6) - 2, .975) and 0.8485527590372793 = (532.4767998709505 - 80.6421422171142) / 532.4767998709505 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in MicroBenchmarks.Serializers.Json_FromString<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 6.60 ms 6.19 ms 0.94 0.00 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;CollectionsOfPrimitives&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;CollectionsOfPrimitives&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;CollectionsOfPrimitives&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;CollectionsOfPrimitives&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromString<CollectionsOfPrimitives>.SystemTextJson_SourceGen_

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 16.26918235514919 (T) = (0 -6191322.1245802445) / Math.Sqrt((14462711183.848114 / (89)) + (2349655928.314376 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.058252612690949473 = (6574291.798431564 - 6191322.1245802445) / 6574291.798431564 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.ReadJson<ImmutableSortedDictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromStream - Duration of single invocation 968.72 μs 910.42 μs 0.94 0.03 False
DeserializeFromString - Duration of single invocation 996.64 μs 927.05 μs 0.93 0.02 False
DeserializeFromString - Duration of single invocation 1.00 ms 930.19 μs 0.93 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<ImmutableSortedDictionary<String, String>>.DeserializeFromStream(Mode: SourceGen)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 18.041284647063847 (T) = (0 -916853.8885510935) / Math.Sqrt((500663616.93650436 / (89)) + (13039281.999953564 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.05208883146454923 = (967236.0860223415 - 916853.8885510935) / 967236.0860223415 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 15.860430468663282 (T) = (0 -930221.3461420626) / Math.Sqrt((370691213.1294784 / (89)) + (106347387.36054114 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.07387815659696739 = (1004426.5263455685 - 930221.3461420626) / 1004426.5263455685 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 22.50496381082304 (T) = (0 -929471.2412983463) / Math.Sqrt((410439909.71599805 / (89)) + (34934721.026406236 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.07253818517464336 = (1002166.5867433777 - 929471.2412983463) / 1002166.5867433777 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ConditionalSelectBenchmark - Duration of single invocation 89.77 ns 15.27 ns 0.17 0.26 False
AddBenchmark - Duration of single invocation 406.43 ns 10.62 ns 0.03 0.21 False
EqualsStaticBenchmark - Duration of single invocation 451.31 ns 11.43 ns 0.03 0.22 False
AndNotBenchmark - Duration of single invocation 65.47 ns 11.50 ns 0.18 0.40 False
OnesComplementOperatorBenchmark - Duration of single invocation 43.22 ns 8.85 ns 0.20 0.47 False
UnaryNegateOperatorBenchmark - Duration of single invocation 429.40 ns 7.69 ns 0.02 0.26 False
SubtractBenchmark - Duration of single invocation 461.22 ns 11.90 ns 0.03 0.20 False
EqualsAllBenchmark - Duration of single invocation 55.08 ns 9.61 ns 0.17 0.37 False
BitwiseAndBenchmark - Duration of single invocation 65.55 ns 11.02 ns 0.17 0.35 False
BitwiseOrBenchmark - Duration of single invocation 64.23 ns 11.14 ns 0.17 0.41 False
XorBenchmark - Duration of single invocation 61.40 ns 11.87 ns 0.19 0.35 False
OnesComplementBenchmark - Duration of single invocation 55.68 ns 7.58 ns 0.14 0.46 False
SubtractionOperatorBenchmark - Duration of single invocation 419.69 ns 11.26 ns 0.03 0.20 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.ConditionalSelectBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 53.63770315034236 (T) = (0 -15.48929369427085) / Math.Sqrt((187.45261511358248 / (98)) + (1.5697507485645503 / (6))) is greater than 1.9834952585625159 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (98) + (6) - 2, .975) and 0.8362355413134266 = (94.58275512585799 - 15.48929369427085) / 94.58275512585799 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 225.84181629995342 (T) = (0 -11.30751310196692) / Math.Sqrt((341.6299794200777 / (99)) + (0.2955142607005624 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9739351621744681 = (433.8224997851555 - 11.30751310196692) / 433.8224997851555 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.EqualsStaticBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 181.18798664418554 (T) = (0 -11.184178300244204) / Math.Sqrt((619.9949865843209 / (99)) + (0.07297639540082072 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9759505750364189 = (465.04971811928056 - 11.184178300244204) / 465.04971811928056 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 44.389894415670526 (T) = (0 -11.562994941349494) / Math.Sqrt((159.40690277992388 / (99)) + (0.7960828384405751 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8352033765456669 = (70.16524185371871 - 11.562994941349494) / 70.16524185371871 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 30.82369283716853 (T) = (0 -7.855159584462673) / Math.Sqrt((192.46633458440166 / (99)) + (0.49332377039426323 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8481578106888099 = (51.73239150526253 - 7.855159584462673) / 51.73239150526253 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 212.25252316158262 (T) = (0 -7.777372095197183) / Math.Sqrt((334.652660237679 / (90)) + (0.3163250480527072 / (6))) is greater than 1.98552344186624 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (90) + (6) - 2, .975) and 0.9814805273540796 = (419.9564557746978 - 7.777372095197183) / 419.9564557746978 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 240.69958343626476 (T) = (0 -11.543076051420401) / Math.Sqrt((274.3811416730374 / (90)) + (0.13670339586386074 / (6))) is greater than 1.98552344186624 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (90) + (6) - 2, .975) and 0.9733652040756469 = (433.38331122207717 - 11.543076051420401) / 433.38331122207717 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 92.62408461410739 (T) = (0 -9.747950645723407) / Math.Sqrt((21.899310577288077 / (99)) + (0.09831705702145851 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8224288417963658 = (54.89602446893259 - 9.747950645723407) / 54.89602446893259 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 40.8885482800187 (T) = (0 -11.759196172728643) / Math.Sqrt((167.27923805767983 / (99)) + (1.1395030824553765 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8266037002940584 = (67.81688071008877 - 11.759196172728643) / 67.81688071008877 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 38.50468861166305 (T) = (0 -11.820339365381146) / Math.Sqrt((202.4160789217082 / (99)) + (0.4657579114868155 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8259501621931051 = (67.91353278074061 - 11.820339365381146) / 67.91353278074061 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 41.75666566005778 (T) = (0 -11.371915684903586) / Math.Sqrt((178.7658663863254 / (99)) + (0.17966071613243234 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8326344214381814 = (67.9465621463091 - 11.371915684903586) / 67.9465621463091 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 32.5405851843065 (T) = (0 -7.495447256530213) / Math.Sqrt((183.96473635852394 / (99)) + (0.02266776162569742 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8555758033388005 = (51.89883295050318 - 7.495447256530213) / 51.89883295050318 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 229.40456570982525 (T) = (0 -11.245423374786531) / Math.Sqrt((306.9572943154058 / (92)) + (0.21222606375573477 / (6))) is greater than 1.9849843115220658 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (92) + (6) - 2, .975) and 0.9739985783411969 = (432.49263530093293 - 11.245423374786531) / 432.49263530093293 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfAny - Duration of single invocation 622.46 ns 418.07 ns 0.67 0.21 False
Split - Duration of single invocation 758.68 ns 585.21 ns 0.77 0.09 False
Replace_String - Duration of single invocation 6.24 μs 628.63 ns 0.10 0.06 False
Replace_Char - Duration of single invocation 2.96 μs 401.87 ns 0.14 0.07 False
Replace_Char - Duration of single invocation 9.06 μs 625.67 ns 0.07 0.07 False
Replace_Char - Duration of single invocation 100.35 μs 3.39 μs 0.03 0.15 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_String*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_String*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_String*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_String*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.IndexOfAny

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 32.94226726279577 (T) = (0 -405.73739708030536) / Math.Sqrt((965.037805435065 / (109)) + (118.786671010087 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.30293729168813993 = (582.0672835345279 - 405.73739708030536) / 582.0672835345279 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 25.335350600894916 (T) = (0 -594.2392242654395) / Math.Sqrt((1765.6717689013642 / (109)) + (61.50658634060228 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.17983663301400493 = (724.5376326050741 - 594.2392242654395) / 724.5376326050741 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "b")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 40.026409499129024 (T) = (0 -639.1345741264005) / Math.Sqrt((1927750.2914931953 / (109)) + (89.86249835361971 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.8928420577293215 = (5964.416268016426 - 639.1345741264005) / 5964.416268016426 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'i', newChar: 'I')

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 39.8296987292001 (T) = (0 -402.403145276744) / Math.Sqrt((389476.6052622854 / (109)) + (24.057515017533504 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.8554898830515266 = (2784.6018934454605 - 402.403145276744) / 2784.6018934454605 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58l", oldChar: 'b', newChar: '+')

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 40.76726140011915 (T) = (0 -629.7826195255576) / Math.Sqrt((3952755.1114664404 / (109)) + (69.02318417528049 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.9249753924230615 = (8394.347399680946 - 629.7826195255576) / 8394.347399680946 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58lzfdql1fehvs91yzkt9xam7ahjbhvpd9edll13ab46i74ktwwgkgbi792e5gkuuzevo5qm8qt83edag7zovoe686gmtw730kms2i5xgji4xcp25287q68fvhwszd3mszht2uh7bchlgkj5qnq1x9m4lg7vwn8cq5l756akua6oyx9k71bmxbysnmhvxvlxde4k9maumfgxd8gxhxx4mwpph2ttyox9zilt3ylv1q9s4bopfuoa8qlrzodg2q67sh85wx4slcd6w7ufnendaxai633ove2ktbaxdt2sz6y6mo42473xd274gz833p6hj3mu77c4m4od9e5s8btxleh0efqnu9zj9rwtbk5758lio35b3q426j5fwwq1qyknfedrsmqyfw1m38mkkotdf7n0vr6p3erhy8dkzntr9fwjrslxjgrbegih0n6bpb5bfuy55bu65ce9kejcfifxwpcs05umrsb8kvd64q2iwugbbi7vd35g5ho0rff9rhombgzzaniyq7bbjbqr88jyw4ccgnoyl31of3a5thv0vg08gnrqzxas800hewtw8tnwgw5pav81ntdpdd62689x3iqpc317y82b3e2trbpdzieoxldaz009tz37gqmh4bdp1bv9lnl5s58udb11z0h7i2sdl5nbyhjyfzxwzezmp4qx0i3eyvsd3fg8sryq9jhlvkonnfcvb4snl4mcbimdzg49tzdhqjmfxfcq3p1st6b9x2xyevo17evpqp4yc4f2rm0f26ivr3t2f5m0boc44vituxaovcqy1jrkcs6im2kdu3jvcexx2k76egve63aon5a6nbxss4rcke90npmqp35qluf571ms160y2nhaqef835wah41qru8tauu362v0r8konl8", oldChar: 'b', newChar: '+')

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/9/2022 1:30:37 AM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 40.2557146147606 (T) = (0 -3325.7331207323646) / Math.Sqrt((570758640.7654835 / (109)) + (19383.138572780637 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.9651651022333828 = (95471.304179324 - 3325.7331207323646) / 95471.304179324 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.ReadJson<Hashtable>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromString - Duration of single invocation 588.83 μs 519.52 μs 0.88 0.02 False
DeserializeFromReader - Duration of single invocation 697.19 μs 610.93 μs 0.88 0.03 False
DeserializeFromString - Duration of single invocation 588.75 μs 523.76 μs 0.89 0.02 False
DeserializeFromUtf8Bytes - Duration of single invocation 533.15 μs 491.60 μs 0.92 0.03 False
DeserializeFromStream - Duration of single invocation 548.01 μs 508.65 μs 0.93 0.02 False
DeserializeFromReader - Duration of single invocation 673.29 μs 614.20 μs 0.91 0.02 False
DeserializeFromStream - Duration of single invocation 549.42 μs 503.89 μs 0.92 0.06 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<Hashtable>.DeserializeFromString(Mode: Reflection)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 26.555207094385505 (T) = (0 -524970.6005487926) / Math.Sqrt((103007342.05823441 / (89)) + (35905250.22007639 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.11908240087806814 = (595936.1023914894 - 524970.6005487926) / 595936.1023914894 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 22.216596285289484 (T) = (0 -617151.1130990243) / Math.Sqrt((180020773.1701634 / (89)) + (33516026.102528572 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.09032874652837061 = (678433.1270706378 - 617151.1130990243) / 678433.1270706378 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 55.54177799066832 (T) = (0 -522779.7506241733) / Math.Sqrt((97973514.8705812 / (89)) + (3957331.3695294545 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.12354720301540786 = (596472.2257978757 - 522779.7506241733) / 596472.2257978757 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 18.957961137006773 (T) = (0 -496494.9626767614) / Math.Sqrt((133511443.0107196 / (89)) + (17903648.346069843 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.07480748955070926 = (536639.62588246 - 496494.9626767614) / 536639.62588246 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 33.924803906326176 (T) = (0 -509846.38775006117) / Math.Sqrt((93450669.58061479 / (89)) + (3832638.69416555 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.07958788209018798 = (553932.7197341607 - 509846.38775006117) / 553932.7197341607 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 10.448817065290026 (T) = (0 -621931.992436866) / Math.Sqrt((132602561.8580844 / (89)) + (163479936.507275 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.08262101029236335 = (677944.4476214483 - 621931.992436866) / 677944.4476214483 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 19.97075390462342 (T) = (0 -509731.50211336603) / Math.Sqrt((131984437.83777577 / (89)) + (21270270.076142002 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.0807572521250231 = (554512.3997896287 - 509731.50211336603) / 554512.3997896287 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in MicroBenchmarks.Serializers.Json_ToString<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_Reflection_ - Duration of single invocation 4.43 ms 3.90 ms 0.88 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;CollectionsOfPrimitives&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;CollectionsOfPrimitives&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;CollectionsOfPrimitives&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;CollectionsOfPrimitives&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_ToString<CollectionsOfPrimitives>.SystemTextJson_Reflection_

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 15.334001452022132 (T) = (0 -3887279.6692708335) / Math.Sqrt((83866416291.12737 / (89)) + (147161849.88427958 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.10925511104580536 = (4364077.433927024 - 3887279.6692708335) / 4364077.433927024 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.WriteJson<HashSet<String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 248.27 μs 225.65 μs 0.91 0.03 False
SerializeObjectProperty - Duration of single invocation 239.16 μs 213.68 μs 0.89 0.03 False
SerializeObjectProperty - Duration of single invocation 253.86 μs 228.55 μs 0.90 0.03 False
SerializeToString - Duration of single invocation 232.62 μs 208.70 μs 0.90 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;HashSet&lt;String&gt;&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;HashSet&lt;String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;HashSet&lt;String&gt;&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;HashSet&lt;String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<HashSet<String>>.SerializeToString(Mode: Reflection)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 19.029960073450354 (T) = (0 -224693.61030792582) / Math.Sqrt((53155599.14466803 / (89)) + (7513328.034697572 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.10328256861270832 = (250573.48328816058 - 224693.61030792582) / 250573.48328816058 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;HashSet&lt;String&gt;&gt;.SerializeObjectProperty(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 35.58934211533524 (T) = (0 -212908.21089093908) / Math.Sqrt((55058839.99652414 / (89)) + (443714.68130461127 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.12212354523072647 = (242526.3939296519 - 212908.21089093908) / 242526.3939296519 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;HashSet&lt;String&gt;&gt;.SerializeObjectProperty(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 31.757270726621844 (T) = (0 -228208.6199651924) / Math.Sqrt((54050606.8552857 / (89)) + (1310372.5187342414 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.1122564421282455 = (257065.92623695495 - 228208.6199651924) / 257065.92623695495 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;HashSet&lt;String&gt;&gt;.SerializeToString(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 30.666056843795506 (T) = (0 -208504.2087587719) / Math.Sqrt((53021578.735908635 / (89)) + (1049478.472124636 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.11435018722536415 = (235425.1147025628 - 208504.2087587719) / 235425.1147025628 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.IO.Tests.BinaryWriterExtendedTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteAsciiCharArray - Duration of single invocation 53.43 μs 32.14 μs 0.60 0.42 False
WriteAsciiString - Duration of single invocation 55.28 μs 30.28 μs 0.55 0.42 False
WriteAsciiString - Duration of single invocation 23.98 ms 15.41 ms 0.64 0.23 False
WriteAsciiCharArray - Duration of single invocation 15.82 ms 8.04 ms 0.51 0.38 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.BinaryWriterExtendedTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.BinaryWriterExtendedTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.BinaryWriterExtendedTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.BinaryWriterExtendedTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 8000)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 32.786182781371394 (T) = (0 -31567.122786902506) / Math.Sqrt((90512658.17896947 / (109)) + (1059579.3939515292 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.5103417299903353 = (64467.65983607189 - 31567.122786902506) / 64467.65983607189 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString(StringLengthInChars: 8000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 29.243246117001874 (T) = (0 -31222.140844712136) / Math.Sqrt((102430191.73213941 / (109)) + (2529341.8580897474 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.5221692463157662 = (65341.421840220755 - 31222.140844712136) / 65341.421840220755 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString(StringLengthInChars: 2000000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 52.309796615985675 (T) = (0 -15745292.981415832) / Math.Sqrt((4069835210673.081 / (109)) + (30316253649.641926 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.4061819946214876 = (26515351.23354746 - 15745292.981415832) / 26515351.23354746 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 2000000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 60.07462310624318 (T) = (0 -8011053.490671742) / Math.Sqrt((3080955328390.025 / (109)) + (11010282060.668827 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.5654133429694176 = (18433730.905152928 - 8011053.490671742) / 18433730.905152928 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Float

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseOrOperatorBenchmark - Duration of single invocation 61.81 ns 12.22 ns 0.20 0.39 False
BitwiseAndOperatorBenchmark - Duration of single invocation 62.40 ns 11.19 ns 0.18 0.36 False
BitwiseOrBenchmark - Duration of single invocation 59.90 ns 10.89 ns 0.18 0.33 False
BitwiseAndBenchmark - Duration of single invocation 62.19 ns 10.85 ns 0.17 0.38 False
ConditionalSelectBenchmark - Duration of single invocation 91.30 ns 14.63 ns 0.16 0.28 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseOrOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 40.29750815326799 (T) = (0 -11.586291684325724) / Math.Sqrt((181.52759387776803 / (99)) + (0.9537785343133743 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8307821839631473 = (68.46969164170297 - 11.586291684325724) / 68.46969164170297 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 39.39606823008167 (T) = (0 -11.386281048218246) / Math.Sqrt((192.32714028835957 / (99)) + (0.7946259510429574 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8328929035950888 = (68.13762726526322 - 11.386281048218246) / 68.13762726526322 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 46.76049948661194 (T) = (0 -11.302204911230342) / Math.Sqrt((118.97764276922318 / (92)) + (0.31204354071687923 / (6))) is greater than 1.9849843115220658 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (92) + (6) - 2, .975) and 0.8275451401485672 = (65.53717837216655 - 11.302204911230342) / 65.53717837216655 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 41.33104487262801 (T) = (0 -11.460101301692065) / Math.Sqrt((176.41666776993944 / (99)) + (0.7608015264214348 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8328519970232384 = (68.5625978031299 - 11.460101301692065) / 68.5625978031299 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 62.90406814402295 (T) = (0 -15.124897320615942) / Math.Sqrt((115.83663354804833 / (90)) + (1.3723108834937245 / (6))) is greater than 1.98552344186624 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (90) + (6) - 2, .975) and 0.8366126915734922 = (92.5708212362111 - 15.124897320615942) / 92.5708212362111 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.IO.Tests.Perf_Path

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetFullPathWithRedundantSegments - Duration of single invocation 19.36 μs 17.07 μs 0.88 0.07 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_Path*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.Perf_Path*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_Path*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.Perf_Path*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_Path.GetFullPathWithRedundantSegments

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 24.31068548543662 (T) = (0 -17070.158165497527) / Math.Sqrt((370690.15372156963 / (103)) + (50493.69996839699 / (6))) is greater than 1.9823833701754254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (103) + (6) - 2, .975) and 0.13502574953653346 = (19734.87436920934 - 17070.158165497527) / 19734.87436920934 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Int

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AndNotBenchmark - Duration of single invocation 63.50 ns 11.13 ns 0.18 0.36 False
MultiplyBenchmark - Duration of single invocation 190.31 ns 11.14 ns 0.06 0.23 False
BitwiseAndOperatorBenchmark - Duration of single invocation 62.10 ns 11.32 ns 0.18 0.41 False
SubtractionOperatorBenchmark - Duration of single invocation 171.24 ns 11.04 ns 0.06 0.26 False
EqualsBenchmark - Duration of single invocation 61.44 ns 10.33 ns 0.17 0.36 False
AddBenchmark - Duration of single invocation 165.69 ns 11.28 ns 0.07 0.19 False
OnesComplementOperatorBenchmark - Duration of single invocation 55.70 ns 7.42 ns 0.13 0.49 False
SubtractBenchmark - Duration of single invocation 162.93 ns 11.14 ns 0.07 0.31 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 62.36 ns 11.29 ns 0.18 0.35 False
NegateBenchmark - Duration of single invocation 161.27 ns 7.75 ns 0.05 0.21 False
BitwiseOrOperatorBenchmark - Duration of single invocation 78.04 ns 10.88 ns 0.14 0.32 False
OnesComplementBenchmark - Duration of single invocation 47.09 ns 7.39 ns 0.16 0.50 False
MultiplyOperatorBenchmark - Duration of single invocation 171.31 ns 10.96 ns 0.06 0.28 False
EqualityOperatorBenchmark - Duration of single invocation 48.90 ns 9.28 ns 0.19 0.43 False
XorBenchmark - Duration of single invocation 59.74 ns 10.78 ns 0.18 0.38 False
BitwiseAndBenchmark - Duration of single invocation 58.13 ns 11.23 ns 0.19 0.40 False
InequalityOperatorBenchmark - Duration of single invocation 46.54 ns 9.23 ns 0.20 0.43 False
UnaryNegateOperatorBenchmark - Duration of single invocation 159.67 ns 7.55 ns 0.05 0.35 False
AddOperatorBenchmark - Duration of single invocation 193.71 ns 12.40 ns 0.06 0.22 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.AndNotBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/3/2022 11:38:23 AM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 42.75637156024062 (T) = (0 -11.028630837126146) / Math.Sqrt((195.63887555641728 / (99)) + (0.014370843438423531 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8450383698633234 = (71.17007498823327 - 11.028630837126146) / 71.17007498823327 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.MultiplyBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 86.43673424265235 (T) = (0 -11.808914994959524) / Math.Sqrt((376.3877974370963 / (99)) + (0.8318159079736909 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9356085301650156 = (183.39253670124575 - 11.808914994959524) / 183.39253670124575 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 39.891370538132556 (T) = (0 -11.201200540073474) / Math.Sqrt((207.91118530820583 / (99)) + (0.027094048412715243 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8378352353596468 = (69.07296147171884 - 11.201200540073474) / 69.07296147171884 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 90.21597195367498 (T) = (0 -11.037647674936695) / Math.Sqrt((350.11770050430715 / (99)) + (0.022726292456102644 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9389463026682805 = (180.7859008925617 - 11.037647674936695) / 180.7859008925617 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 88.50941517330601 (T) = (0 -10.50951847289617) / Math.Sqrt((33.84849677892461 / (90)) + (0.05090760493544301 / (6))) is greater than 1.98552344186624 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (90) + (6) - 2, .975) and 0.8392994084310924 = (65.39813183195248 - 10.50951847289617) / 65.39813183195248 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 82.86144586366046 (T) = (0 -12.009821540582129) / Math.Sqrt((393.9097812313969 / (99)) + (1.6731217795407833 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9343684546600095 = (182.988553421495 - 12.009821540582129) / 182.988553421495 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 32.6000412412601 (T) = (0 -7.596316699646363) / Math.Sqrt((179.94131770199698 / (99)) + (0.01425502579624428 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8527153498973283 = (51.575752764127145 - 7.596316699646363) / 51.575752764127145 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 86.23409752530553 (T) = (0 -11.049341643110518) / Math.Sqrt((401.98466404868 / (99)) + (0.02281280851202335 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9402406299530376 = (184.89722422487569 - 11.049341643110518) / 184.89722422487569 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 42.32604645244054 (T) = (0 -11.122401312517058) / Math.Sqrt((176.46809874005484 / (99)) + (0.020206592001493173 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8356752399035132 = (67.68548638675209 - 11.122401312517058) / 67.68548638675209 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 83.9569718413585 (T) = (0 -7.506406417079617) / Math.Sqrt((383.80067028294894 / (99)) + (0.025313689402383162 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9565861840499773 = (172.90363108649285 - 7.506406417079617) / 172.90363108649285 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 41.32869998221179 (T) = (0 -11.010173790691004) / Math.Sqrt((187.58635738086488 / (99)) + (0.028432713971590706 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8380168746014953 = (67.97111590237685 - 11.010173790691004) / 67.97111590237685 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 32.986925041252235 (T) = (0 -7.412145596862225) / Math.Sqrt((173.606357878002 / (99)) + (0.021006881497971593 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8550565433558288 = (51.138187045301834 - 7.412145596862225) / 51.138187045301834 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.MultiplyOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 81.94442344858284 (T) = (0 -11.182862547432537) / Math.Sqrt((424.8172192378537 / (99)) + (0.06093905592642177 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9382608876313698 = (181.13092524981903 - 11.182862547432537) / 181.13092524981903 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 90.73969945752015 (T) = (0 -9.281995895487896) / Math.Sqrt((21.49131148834626 / (98)) + (0.03347356639969336 / (6))) is greater than 1.9834952585625159 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (98) + (6) - 2, .975) and 0.8225645085566807 = (52.31194627402361 - 9.281995895487896) / 52.31194627402361 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 41.36760462399945 (T) = (0 -11.011038464162496) / Math.Sqrt((186.46620936613084 / (99)) + (0.05223593132318436 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8378708462863989 = (67.91522814961051 - 11.011038464162496) / 67.91522814961051 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 39.84128238381385 (T) = (0 -11.543437937023294) / Math.Sqrt((188.84115916890124 / (99)) + (0.7237730043162 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8309450637928478 = (68.28217025782965 - 11.543437937023294) / 68.28217025782965 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.InequalityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 76.89257244057006 (T) = (0 -9.259608900014937) / Math.Sqrt((28.66399821320982 / (99)) + (0.0102686086351388 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8175678078432169 = (50.75644156080297 - 9.259608900014937) / 50.75644156080297 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 78.48666956029705 (T) = (0 -7.706859078374726) / Math.Sqrt((422.453305074888 / (99)) + (0.3949132396333267 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.954952792507431 = (171.08405842129184 - 7.706859078374726) / 171.08405842129184 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 83.75660968398918 (T) = (0 -11.81271074023032) / Math.Sqrt((382.58611678598413 / (99)) + (0.8871955163730101 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9342220987589112 = (179.58479242039724 - 11.81271074023032) / 179.58479242039724 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.ReadJson<HashSet<String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromString - Duration of single invocation 246.66 μs 208.45 μs 0.85 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 221.78 μs 195.13 μs 0.88 0.03 False
DeserializeFromString - Duration of single invocation 246.71 μs 215.43 μs 0.87 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<HashSet<String>>.DeserializeFromString(Mode: SourceGen)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 34.40286084184879 (T) = (0 -209940.2820100309) / Math.Sqrt((82167673.95955402 / (89)) + (896787.9742712888 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.14509577587939393 = (245571.6980764543 - 209940.2820100309) / 245571.6980764543 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 14.745805893195215 (T) = (0 -194800.7439588727) / Math.Sqrt((143118741.01610237 / (89)) + (1552347.847447059 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.09373114014194145 = (214948.07179999846 - 194800.7439588727) / 214948.07179999846 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 21.64520426130609 (T) = (0 -211499.6458470916) / Math.Sqrt((80971440.32405281 / (89)) + (9260867.801890466 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.13815149811459582 = (245402.34784235162 - 211499.6458470916) / 245402.34784235162 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 7.34 ms 4.30 ms 0.59 0.22 False
Count - Duration of single invocation 6.89 ms 4.48 ms 0.65 0.19 False
Count - Duration of single invocation 7.22 ms 4.86 ms 0.67 0.15 False
Count - Duration of single invocation 24.77 ms 23.04 ms 0.93 0.01 False
Count - Duration of single invocation 9.18 ms 6.58 ms 0.72 0.14 False
Count - Duration of single invocation 25.20 ms 22.33 ms 0.89 0.02 False
Count - Duration of single invocation 8.48 ms 5.30 ms 0.62 0.21 False
Count - Duration of single invocation 12.49 ms 9.64 ms 0.77 0.07 False
Count - Duration of single invocation 7.11 ms 4.61 ms 0.65 0.16 False
Count - Duration of single invocation 10.49 ms 7.94 ms 0.76 0.09 False
Count - Duration of single invocation 23.89 ms 21.27 ms 0.89 0.01 False
Count - Duration of single invocation 8.61 ms 5.42 ms 0.63 0.20 False
Count - Duration of single invocation 6.66 ms 4.16 ms 0.62 0.18 False
Count - Duration of single invocation 12.24 ms 9.18 ms 0.75 0.10 False
Count - Duration of single invocation 16.70 ms 14.10 ms 0.84 0.02 False
Count - Duration of single invocation 7.33 ms 4.61 ms 0.63 0.25 False
Count - Duration of single invocation 7.31 ms 4.28 ms 0.58 0.21 False
Count - Duration of single invocation 7.53 ms 4.97 ms 0.66 0.17 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: None)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 74.76147488198862 (T) = (0 -4274725.7996279765) / Math.Sqrt((138792143376.00815 / (108)) + (1056089431.7835646 / (6))) is greater than 1.9813718148759247 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (108) + (6) - 2, .975) and 0.40066603193544426 = (7132460.41006562 - 4274725.7996279765) / 7132460.41006562 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 72.38825740546316 (T) = (0 -4404203.569928164) / Math.Sqrt((71755665444.74855 / (109)) + (2477908171.3772535 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.34979064836013973 = (6773516.189548095 - 4404203.569928164) / 6773516.189548095 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sher[a-z]+|Hol[a-z]+", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 36.43852651249812 (T) = (0 -4798664.16400145) / Math.Sqrt((80360263171.25722 / (109)) + (18684915212.32785 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.32030513530189103 = (7060027.09926727 - 4798664.16400145) / 7060027.09926727 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the\\s+\\w+", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/26/2022 8:24:36 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 17.47646842386191 (T) = (0 -22747785.551060054) / Math.Sqrt((381347365191.54675 / (105)) + (83186677727.1038 / (6))) is greater than 1.9819674897361885 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (105) + (6) - 2, .975) and 0.09224766007938746 = (25059462.312208924 - 22747785.551060054) / 25059462.312208924 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sher[a-z]+|Hol[a-z]+", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 39.021172514694605 (T) = (0 -6676822.420557804) / Math.Sqrt((100464697623.64671 / (109)) + (15380024360.461573 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.2565120626214106 = (8980404.502726879 - 6676822.420557804) / 8980404.502726879 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the\\s+\\w+", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/25/2022 11:45:49 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 11.7862669783192 (T) = (0 -22711971.08189033) / Math.Sqrt((392943220258.9158 / (105)) + (213718042633.31055 / (6))) is greater than 1.9819674897361885 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (105) + (6) - 2, .975) and 0.09334713832777368 = (25050349.524073057 - 22711971.08189033) / 25050349.524073057 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 56.69297284672361 (T) = (0 -5406526.632638889) / Math.Sqrt((122048869184.39806 / (109)) + (8894745121.41042 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.34849504111255314 = (8298519.541388347 - 5406526.632638889) / 8298519.541388347 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes.{0,25}Watson|Watson.{0,25}Holmes", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 32.020903798864914 (T) = (0 -9914155.409487586) / Math.Sqrt((160727720358.28433 / (109)) + (22447119097.85811 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.18913959529390717 = (12226710.481788915 - 9914155.409487586) / 12226710.481788915 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes.{0,25}Watson|Watson.{0,25}Holmes", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 56.47980559395981 (T) = (0 -4686525.524920327) / Math.Sqrt((58545852532.35773 / (109)) + (7644131826.011393 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.33901030123621195 = (7090164.239602027 - 4686525.524920327) / 7090164.239602027 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes.{0,25}Watson|Watson.{0,25}Holmes", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 43.84515628647385 (T) = (0 -7934611.729985547) / Math.Sqrt((114651807929.1577 / (109)) + (10669339345.546368 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.22718127239856478 = (10267105.915784243 - 7934611.729985547) / 10267105.915784243 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 22.71579744744644 (T) = (0 -21581645.587980997) / Math.Sqrt((769902483880.4392 / (109)) + (47211127066.39357 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.11396025123179508 = (24357423.713760417 - 21581645.587980997) / 24357423.713760417 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 77.0109356513666 (T) = (0 -5451146.177825616) / Math.Sqrt((110705214932.92195 / (109)) + (3018531184.6761518 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.3550720266025175 = (8452333.287869282 - 5451146.177825616) / 8452333.287869282 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 37.24478983256799 (T) = (0 -4169600.9182525375) / Math.Sqrt((77909327917.61078 / (109)) + (18665512576.238 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.35587405927512733 = (6473269.673878126 - 4169600.9182525375) / 6473269.673878126 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 29.65751395620889 (T) = (0 -9336365.90304572) / Math.Sqrt((184504613279.22495 / (108)) + (41893576231.71483 / (6))) is greater than 1.9813718148759247 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (108) + (6) - 2, .975) and 0.2284725832537174 = (12101145.987033654 - 9336365.90304572) / 12101145.987033654 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 31.811755181120585 (T) = (0 -14275113.350669155) / Math.Sqrt((318911495706.4737 / (109)) + (23676339064.82346 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.15592808745640227 = (16912200.4162552 - 14275113.350669155) / 16912200.4162552 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 33.957086268019225 (T) = (0 -4372750.032311699) / Math.Sqrt((136990044529.22107 / (109)) + (32267948258.16105 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.3874581611826833 = (7138696.0942203 - 4372750.032311699) / 7138696.0942203 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 77.81002710190032 (T) = (0 -4256903.188206993) / Math.Sqrt((134097560482.74847 / (109)) + (826660165.3217188 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.40336502545993513 = (7134853.586966743 - 4256903.188206993) / 7134853.586966743 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sher[a-z]+|Hol[a-z]+", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 33.663503864911654 (T) = (0 -5109374.791826747) / Math.Sqrt((62148871908.291794 / (109)) + (25108634783.99577 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.3123946462295716 = (7430679.188010831 - 5109374.791826747) / 7430679.188010831 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Tests.Perf_Guid

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EqualsOperator - Duration of single invocation 43.14 ns 9.48 ns 0.22 0.38 False
EqualsSame - Duration of single invocation 49.55 ns 10.67 ns 0.22 0.58 False
EqualsNotSame - Duration of single invocation 24.80 ns 9.31 ns 0.38 0.70 False
NotEqualsOperator - Duration of single invocation 51.05 ns 10.41 ns 0.20 0.45 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Guid*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_Guid*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Guid*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_Guid*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Guid.EqualsOperator

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 136.00380694996653 (T) = (0 -9.529678130909767) / Math.Sqrt((10.739786039771253 / (109)) + (0.033018683099658955 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.8215302481529407 = (53.39660100539771 - 9.529678130909767) / 53.39660100539771 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Guid.EqualsSame

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 101.91864536929934 (T) = (0 -10.415194684127941) / Math.Sqrt((11.573985815332204 / (109)) + (0.05324630436439757 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.7684800470171091 = (44.98616447497989 - 10.415194684127941) / 44.98616447497989 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Guid.EqualsNotSame

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 27.458760142516265 (T) = (0 -10.099089438018153) / Math.Sqrt((11.502267169871073 / (109)) + (0.9986065913405717 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.5864210230355238 = (24.418768845897116 - 10.099089438018153) / 24.418768845897116 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Guid.NotEqualsOperator

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 67.50551686652274 (T) = (0 -11.439399563039293) / Math.Sqrt((10.470475711548445 / (102)) + (1.866955601275573 / (6))) is greater than 1.9825972617651593 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (102) + (6) - 2, .975) and 0.7914970312643866 = (54.86444453241682 - 11.439399563039293) / 54.86444453241682 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Tests.Utf8JsonReaderCommentsTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Utf8JsonReaderCommentParsing - Duration of single invocation 182.08 μs 70.74 μs 0.39 0.20 False
Utf8JsonReaderCommentParsing - Duration of single invocation 191.57 μs 74.61 μs 0.39 0.21 False
Utf8JsonReaderCommentParsing - Duration of single invocation 34.28 μs 24.09 μs 0.70 0.18 False
Utf8JsonReaderCommentParsing - Duration of single invocation 18.43 μs 6.79 μs 0.37 0.36 False
Utf8JsonReaderCommentParsing - Duration of single invocation 17.72 μs 6.30 μs 0.36 0.35 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Utf8JsonReaderCommentsTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Utf8JsonReaderCommentsTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Utf8JsonReaderCommentsTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Utf8JsonReaderCommentsTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 0, TestCase: LongMultiLine)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 78.2615640154501 (T) = (0 -72178.03997091974) / Math.Sqrt((264819224.31515592 / (109)) + (3749912.841000583 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.6545798164902726 = (208957.2162157315 - 72178.03997091974) / 208957.2162157315 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 0, TestCase: LongMultiLine)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 71.66805306777276 (T) = (0 -72508.2194224745) / Math.Sqrt((256681692.6359039 / (109)) + (7554347.570688394 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.6526581149941587 = (208751.73007498108 - 72508.2194224745) / 208751.73007498108 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 100, TestCase: LongSingleLine)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/18/2022 12:43:31 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 28.7890157614757 (T) = (0 -24701.287206490106) / Math.Sqrt((4716354.49360174 / (108)) + (695287.5856995157 / (6))) is greater than 1.9813718148759247 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (108) + (6) - 2, .975) and 0.3176577976643073 = (36200.732010912296 - 24701.287206490106) / 36200.732010912296 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 0, TestCase: LongSingleLine)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/18/2022 12:43:31 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 58.28292795402091 (T) = (0 -6532.173278640104) / Math.Sqrt((4121428.707743666 / (109)) + (72517.44061204098 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.6658951711712213 = (19551.268688749475 - 6532.173278640104) / 19551.268688749475 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 0, TestCase: LongSingleLine)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 63.93057731180619 (T) = (0 -6175.700571671249) / Math.Sqrt((4745020.455966387 / (109)) + (4435.013253252941 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.6853498356161402 = (19627.19639369759 - 6175.700571671249) / 19627.19639369759 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in MicroBenchmarks.Serializers.Json_FromStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_Reflection_ - Duration of single invocation 624.41 μs 517.44 μs 0.83 0.03 False
SystemTextJson_SourceGen_ - Duration of single invocation 625.26 μs 521.83 μs 0.83 0.02 False
JsonNet_ - Duration of single invocation 1.01 ms 889.83 μs 0.88 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<IndexViewModel>.SystemTextJson_Reflection_

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 30.499263105335647 (T) = (0 -520775.58983796294) / Math.Sqrt((809183698.1225854 / (89)) + (11879307.284877066 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.16308999298301705 = (622259.9628055291 - 520775.58983796294) / 622259.9628055291 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;.SystemTextJson_SourceGen_

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 30.073706043186014 (T) = (0 -524806.1804232805) / Math.Sqrt((745554160.7475497 / (89)) + (18947871.441681597 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.1629166134170749 = (626946.1189112859 - 524806.1804232805) / 626946.1189112859 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;.JsonNet_

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 4.0221231999207046 (T) = (0 -925635.4546141266) / Math.Sqrt((3058162298.600821 / (100)) + (964189366.7299982 / (6))) is greater than 1.9830375264834914 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (100) + (6) - 2, .975) and 0.05668975147631337 = (981263.0108310371 - 925635.4546141266) / 981263.0108310371 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.WriteJson<ArrayList>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 280.74 μs 257.25 μs 0.92 0.03 False
SerializeToString - Duration of single invocation 284.68 μs 255.13 μs 0.90 0.03 False
SerializeObjectProperty - Duration of single invocation 284.68 μs 258.03 μs 0.91 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ArrayList&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ArrayList&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ArrayList&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ArrayList&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<ArrayList>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 31.792221111418375 (T) = (0 -255912.3094822614) / Math.Sqrt((55520341.19390052 / (89)) + (775324.6351187795 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.09731430193197775 = (283501.0126226427 - 255912.3094822614) / 283501.0126226427 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;ArrayList&gt;.SerializeToString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 33.7289332663225 (T) = (0 -254950.59547491043) / Math.Sqrt((53959360.21662141 / (89)) + (402450.2892134999 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.09792892519722854 = (282628.05736305506 - 254950.59547491043) / 282628.05736305506 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;ArrayList&gt;.SerializeObjectProperty(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 34.411101030677926 (T) = (0 -258363.86256280149) / Math.Sqrt((58656433.23685081 / (89)) + (282467.6915484585 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.1006555980702178 = (287280.22547136917 - 258363.86256280149) / 287280.22547136917 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SubtractBenchmark - Duration of single invocation 427.46 ns 10.75 ns 0.03 0.23 False
AddBenchmark - Duration of single invocation 427.57 ns 11.19 ns 0.03 0.24 False
OnesComplementBenchmark - Duration of single invocation 47.02 ns 7.61 ns 0.16 0.42 False
EqualsAllBenchmark - Duration of single invocation 51.18 ns 9.01 ns 0.18 0.31 False
ConditionalSelectBenchmark - Duration of single invocation 91.01 ns 16.80 ns 0.18 0.22 False
EqualsStaticBenchmark - Duration of single invocation 449.57 ns 11.03 ns 0.02 0.18 False
XorBenchmark - Duration of single invocation 58.50 ns 11.10 ns 0.19 0.31 False
AndNotBenchmark - Duration of single invocation 63.87 ns 10.79 ns 0.17 0.42 False
UnaryNegateOperatorBenchmark - Duration of single invocation 442.63 ns 7.49 ns 0.02 0.27 False
GreaterThanBenchmark - Duration of single invocation 442.77 ns 11.23 ns 0.03 0.23 False
SubtractionOperatorBenchmark - Duration of single invocation 419.15 ns 11.81 ns 0.03 0.23 False
BitwiseOrBenchmark - Duration of single invocation 62.97 ns 11.00 ns 0.17 0.39 False
BitwiseAndBenchmark - Duration of single invocation 62.24 ns 12.17 ns 0.20 0.42 False
OnesComplementOperatorBenchmark - Duration of single invocation 43.66 ns 7.30 ns 0.17 0.50 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.SubtractBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 219.75358367919753 (T) = (0 -10.931699964978108) / Math.Sqrt((335.00353838639893 / (90)) + (0.04832907770230188 / (6))) is greater than 1.98552344186624 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (90) + (6) - 2, .975) and 0.9748906873036397 = (435.3643644958362 - 10.931699964978108) / 435.3643644958362 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 209.56344994702053 (T) = (0 -11.224350856107316) / Math.Sqrt((410.08376546538767 / (99)) + (0.008305500648191511 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9743625381208089 = (437.8105332344799 - 11.224350856107316) / 437.8105332344799 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 32.36938636248365 (T) = (0 -7.4877080351182395) / Math.Sqrt((186.7345527601769 / (99)) + (0.02109367918193388 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8559640614796445 = (51.98499841107404 - 7.4877080351182395) / 51.98499841107404 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 87.19390225454221 (T) = (0 -9.491061330168767) / Math.Sqrt((17.353651481743988 / (99)) + (0.5311515967370058 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8251345657009397 = (54.27637181821115 - 9.491061330168767) / 54.27637181821115 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 55.607972133375064 (T) = (0 -15.03231406503499) / Math.Sqrt((190.7483383123939 / (98)) + (0.8021079532628559 / (6))) is greater than 1.9834952585625159 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (98) + (6) - 2, .975) and 0.8421524696505609 = (95.23312801763079 - 15.03231406503499) / 95.23312801763079 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.EqualsStaticBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 192.78533835397218 (T) = (0 -11.13779550491992) / Math.Sqrt((546.1152102700354 / (99)) + (0.18877197952201347 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9760590261640845 = (465.21898320657675 - 11.13779550491992) / 465.21898320657675 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 43.2585874915019 (T) = (0 -11.083318996502257) / Math.Sqrt((163.0407389161254 / (99)) + (0.03686908867120746 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.833835356656549 = (66.70082620159927 - 11.083318996502257) / 66.70082620159927 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 39.19243913177683 (T) = (0 -11.656878204500394) / Math.Sqrt((204.18083437017435 / (99)) + (0.9787984039237904 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8337710810045289 = (70.12545274879629 - 11.656878204500394) / 70.12545274879629 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 229.3601260243782 (T) = (0 -7.615609736172105) / Math.Sqrt((294.1238251470395 / (90)) + (0.2795952074783918 / (6))) is greater than 1.98552344186624 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (90) + (6) - 2, .975) and 0.9820890210768336 = (425.1922672033272 - 7.615609736172105) / 425.1922672033272 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.GreaterThanBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 208.75055750540167 (T) = (0 -11.057865455088196) / Math.Sqrt((428.2368845301636 / (99)) + (0.03643853887260305 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9751801353520653 = (445.52480893598766 - 11.057865455088196) / 445.52480893598766 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 240.71130978646636 (T) = (0 -11.23709637159642) / Math.Sqrt((281.1248778756245 / (92)) + (0.08976891538931953 / (6))) is greater than 1.9849843115220658 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (92) + (6) - 2, .975) and 0.9740508637131178 = (433.0431752087629 - 11.23709637159642) / 433.0431752087629 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 42.02541760248029 (T) = (0 -11.045679768421019) / Math.Sqrt((180.48474565384544 / (99)) + (0.03752356920664186 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8372913224611431 = (67.88623652714016 - 11.045679768421019) / 67.88623652714016 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 42.48606939379433 (T) = (0 -11.262849514127884) / Math.Sqrt((172.71451531478962 / (99)) + (0.27386927035473774 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8346351307036787 = (68.10908243120073 - 11.262849514127884) / 68.10908243120073 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 34.65001282610601 (T) = (0 -7.476985775588452) / Math.Sqrt((157.33146828524661 / (99)) + (0.032203235571099435 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8540557346986722 = (51.23178879383092 - 7.476985775588452) / 51.23178879383092 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Tests.Perf_Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetString - Duration of single invocation 6.04 μs 2.66 μs 0.44 0.22 False
GetString - Duration of single invocation 4.38 μs 1.64 μs 0.37 0.33 False
GetString - Duration of single invocation 497.18 ns 398.52 ns 0.80 0.07 False
GetString - Duration of single invocation 479.06 ns 382.36 ns 0.80 0.03 False
GetBytes - Duration of single invocation 3.79 μs 2.37 μs 0.63 0.46 False
GetBytes - Duration of single invocation 5.88 μs 4.51 μs 0.77 0.30 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Tests.Perf_Encoding*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Tests.Perf_Encoding*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Tests.Perf_Encoding*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Tests.Perf_Encoding*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Tests.Perf_Encoding.GetString(size: 512, encName: "utf-8")

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 8.82039241705438 (T) = (0 -2658.758835979054) / Math.Sqrt((4158301.0430787723 / (109)) + (1078.3850583903456 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.39375301542882385 = (4385.6033986869315 - 2658.758835979054) / 4385.6033986869315 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetString(size: 512, encName: "ascii")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 8.148268421239402 (T) = (0 -1651.9876999326063) / Math.Sqrt((3480409.477271176 / (109)) + (1749.8837626180066 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.46960588165048944 = (3114.641815926032 - 1651.9876999326063) / 3114.641815926032 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetString(size: 16, encName: "utf-8")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 10.386382439344715 (T) = (0 -386.712961146495) / Math.Sqrt((4577.653868445058 / (109)) + (72.4709291246504 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.16492946852809523 = (463.090178101328 - 386.712961146495) / 463.090178101328 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetString(size: 16, encName: "ascii")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 9.957502316783081 (T) = (0 -382.9393754334571) / Math.Sqrt((3855.362516496484 / (109)) + (2.937934336439512 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.13473338023480755 = (442.56806709748656 - 382.9393754334571) / 442.56806709748656 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetBytes(size: 512, encName: "ascii")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 33.439882113150475 (T) = (0 -2459.0134125271184) / Math.Sqrt((316235.30734796147 / (109)) + (4185.304149929105 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.4492777088657864 = (4465.069695041353 - 2459.0134125271184) / 4465.069695041353 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetBytes(size: 512, encName: "utf-8")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 27.656887499034735 (T) = (0 -4591.325879675119) / Math.Sqrt((344862.5027076497 / (109)) + (15086.449418252512 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.3122014969529086 = (6675.393824404362 - 4591.325879675119) / 6675.393824404362 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.IO.Tests.StreamReaderReadLineTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadLineAsync - Duration of single invocation 378.42 μs 192.61 μs 0.51 0.13 False
ReadLineAsync - Duration of single invocation 538.14 μs 391.82 μs 0.73 0.06 False
ReadLineAsync - Duration of single invocation 382.83 μs 209.77 μs 0.55 0.09 False
ReadLineAsync - Duration of single invocation 388.29 μs 204.47 μs 0.53 0.10 False
ReadLine - Duration of single invocation 340.89 μs 169.58 μs 0.50 0.10 False
ReadLineAsync - Duration of single invocation 1.12 ms 980.99 μs 0.88 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StreamReaderReadLineTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StreamReaderReadLineTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StreamReaderReadLineTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StreamReaderReadLineTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [1025, 2048])

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 17.395958019946185 (T) = (0 -197540.73381358435) / Math.Sqrt((4585860680.927044 / (105)) + (66354691.77651246 / (6))) is greater than 1.9819674897361885 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (105) + (6) - 2, .975) and 0.3944930359520778 = (326240.23428729747 - 197540.73381358435) / 326240.23428729747 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [  33,  128])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 14.169504668665388 (T) = (0 -399603.0950649535) / Math.Sqrt((3398560397.3497257 / (109)) + (116926661.59124735 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.2015330773466993 = (500462.929305919 - 399603.0950649535) / 500462.929305919 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 18.859778298945024 (T) = (0 -209067.16893485383) / Math.Sqrt((4501720930.512367 / (103)) + (598862.024959626 / (6))) is greater than 1.9823833701754254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (103) + (6) - 2, .975) and 0.3738487846027951 = (333892.4588723031 - 209067.16893485383) / 333892.4588723031 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 19.317501428136506 (T) = (0 -203895.74224119133) / Math.Sqrt((4557101404.582018 / (109)) + (165595.99821838082 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.37995946970038147 = (328842.60347087955 - 203895.74224119133) / 328842.60347087955 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 20.018240022937132 (T) = (0 -169467.61636531408) / Math.Sqrt((4227737837.2103977 / (109)) + (40467.31210975655 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.4238731001063841 = (294149.8069203277 - 169467.61636531408) / 294149.8069203277 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [   9,   32])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 7.085713529359129 (T) = (0 -1001858.937182826) / Math.Sqrt((2659809657.2662683 / (107)) + (580357482.084831 / (6))) is greater than 1.9815667570746824 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (107) + (6) - 2, .975) and 0.07234402615716651 = (1079989.7434310752 - 1001858.937182826) / 1079989.7434310752 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 7.41 μs 6.82 μs 0.92 0.03 False
SerializeToString - Duration of single invocation 9.11 μs 8.45 μs 0.93 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;LoginViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;LoginViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;LoginViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;LoginViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 15.292165917331502 (T) = (0 -6783.240204413173) / Math.Sqrt((43704.61864199735 / (89)) + (5845.88488829778 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.07944326614188964 = (7368.628086597328 - 6783.240204413173) / 7368.628086597328 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;LoginViewModel&gt;.SerializeToString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 25.225661596713202 (T) = (0 -8451.186175262175) / Math.Sqrt((48997.25412483872 / (89)) + (7.6804873860654705 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.06552228472053712 = (9043.753571731544 - 8451.186175262175) / 9043.753571731544 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
OnesComplementOperatorBenchmark - Duration of single invocation 45.66 ns 7.28 ns 0.16 0.58 False
InequalityOperatorBenchmark - Duration of single invocation 51.12 ns 9.17 ns 0.18 0.42 False
UnaryNegateOperatorBenchmark - Duration of single invocation 253.22 ns 7.66 ns 0.03 0.25 False
OnesComplementBenchmark - Duration of single invocation 43.43 ns 7.57 ns 0.17 0.37 False
ConditionalSelectBenchmark - Duration of single invocation 90.51 ns 14.67 ns 0.16 0.32 False
XorBenchmark - Duration of single invocation 62.22 ns 12.29 ns 0.20 0.34 False
BitwiseAndBenchmark - Duration of single invocation 61.77 ns 10.95 ns 0.18 0.39 False
EqualsStaticBenchmark - Duration of single invocation 292.64 ns 11.14 ns 0.04 0.23 False
AndNotBenchmark - Duration of single invocation 63.71 ns 11.06 ns 0.17 0.38 False
AddBenchmark - Duration of single invocation 266.09 ns 11.09 ns 0.04 0.21 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 62.54 ns 11.11 ns 0.18 0.46 False
MultiplyBenchmark - Duration of single invocation 273.36 ns 11.05 ns 0.04 0.27 False
EqualsAllBenchmark - Duration of single invocation 52.45 ns 9.02 ns 0.17 0.38 False
MultiplyOperatorBenchmark - Duration of single invocation 256.08 ns 11.16 ns 0.04 0.13 False
NegateBenchmark - Duration of single invocation 267.36 ns 7.72 ns 0.03 0.34 False
SubtractBenchmark - Duration of single invocation 269.70 ns 11.31 ns 0.04 0.23 False
BitwiseOrOperatorBenchmark - Duration of single invocation 64.24 ns 10.97 ns 0.17 0.36 False
BitwiseOrBenchmark - Duration of single invocation 62.04 ns 11.36 ns 0.18 0.32 False
EqualityOperatorBenchmark - Duration of single invocation 52.16 ns 9.17 ns 0.18 0.39 False
SubtractionOperatorBenchmark - Duration of single invocation 259.66 ns 11.05 ns 0.04 0.29 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.OnesComplementOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 32.78916510060269 (T) = (0 -7.379072200162741) / Math.Sqrt((178.03240456957985 / (99)) + (0.03389279492974405 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8564904037906917 = (51.41866742764985 - 7.379072200162741) / 51.41866742764985 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 73.65265391548954 (T) = (0 -9.230004592299276) / Math.Sqrt((32.1231157033608 / (99)) + (0.007949848251262224 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.819973241053056 = (51.270181423526395 - 9.230004592299276) / 51.270181423526395 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 82.30838785685448 (T) = (0 -7.594381317098761) / Math.Sqrt((991.6702552928997 / (98)) + (0.006575037143103738 / (6))) is greater than 1.9834952585625159 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (98) + (6) - 2, .975) and 0.971813754807709 = (269.43572176033723 - 7.594381317098761) / 269.43572176033723 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 33.099631704001816 (T) = (0 -7.469851195071179) / Math.Sqrt((170.04219868487758 / (99)) + (0.06912092237456846 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8535165947186941 = (50.99452173934732 - 7.469851195071179) / 50.99452173934732 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 50.20976383223732 (T) = (0 -15.419711169309751) / Math.Sqrt((208.468156416766 / (99)) + (2.588192512413962 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8383598125903805 = (95.39528143600816 - 15.419711169309751) / 95.39528143600816 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 39.38316221782242 (T) = (0 -11.277375804796478) / Math.Sqrt((200.23190930844373 / (99)) + (0.260560754741646 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8338745907198795 = (67.8847134442906 - 11.277375804796478) / 67.8847134442906 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 43.227451103717506 (T) = (0 -10.964427158199149) / Math.Sqrt((175.32694507482626 / (99)) + (0.008988952788181936 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8399705153273669 = (68.51504384100662 - 10.964427158199149) / 68.51504384100662 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.EqualsStaticBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 87.10355713096578 (T) = (0 -11.261103958032669) / Math.Sqrt((1143.1272480047644 / (99)) + (0.11960019069151244 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9633783648134033 = (307.49866576559015 - 11.261103958032669) / 307.49866576559015 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 44.02401787765305 (T) = (0 -10.959675083713906) / Math.Sqrt((175.03777519645445 / (99)) + (0.04943185686941285 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8426100952831967 = (69.63391396312237 - 10.959675083713906) / 69.63391396312237 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 84.36568471373859 (T) = (0 -11.093540987339855) / Math.Sqrt((1010.8437430795484 / (99)) + (0.009039486102918333 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9604783137704653 = (280.69503216311676 - 11.093540987339855) / 280.69503216311676 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 39.82575930432334 (T) = (0 -11.027025592136662) / Math.Sqrt((200.51247970110413 / (99)) + (0.016461570447413212 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8372244524661323 = (67.74374750508724 - 11.027025592136662) / 67.74374750508724 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.MultiplyBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 85.06683816917513 (T) = (0 -11.50771460658153) / Math.Sqrt((984.7245326872328 / (99)) + (0.8373368780835151 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.959144767247939 = (281.6705188394991 - 11.50771460658153) / 281.6705188394991 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 77.75742776883155 (T) = (0 -9.568415108756128) / Math.Sqrt((26.3546672198783 / (99)) + (0.5132220726285208 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8281686290542867 = (55.684913971727994 - 9.568415108756128) / 55.684913971727994 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.MultiplyOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 82.61043909567866 (T) = (0 -11.242898214755897) / Math.Sqrt((1042.9846001348747 / (99)) + (0.02946298489977896 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9597666261976263 = (279.4420937697391 - 11.242898214755897) / 279.4420937697391 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 82.46299541201508 (T) = (0 -7.500958360639676) / Math.Sqrt((1027.838128590176 / (99)) + (0.04028423260979796 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9725535594333803 = (273.2943946750725 - 7.500958360639676) / 273.2943946750725 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 81.89018213684353 (T) = (0 -11.12180589629831) / Math.Sqrt((1079.1768663167081 / (98)) + (0.032247241036593655 / (6))) is greater than 1.9834952585625159 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (98) + (6) - 2, .975) and 0.9606913504656611 = (282.9353342852101 - 11.12180589629831) / 282.9353342852101 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 41.72337643782938 (T) = (0 -11.124959751156398) / Math.Sqrt((184.4618564721842 / (99)) + (0.03704945003277539 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8368105710655117 = (68.17206128972033 - 11.124959751156398) / 68.17206128972033 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 43.83119251318014 (T) = (0 -11.040924755702704) / Math.Sqrt((164.6405715968744 / (99)) + (0.03104906674983478 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8368005658290141 = (67.65295977763626 - 11.040924755702704) / 67.65295977763626 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 68.68989102299973 (T) = (0 -9.72009078285559) / Math.Sqrt((22.666363466849788 / (90)) + (0.6978315814065452 / (6))) is greater than 1.98552344186624 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (90) + (6) - 2, .975) and 0.8108863519373862 = (51.39814541379562 - 9.72009078285559) / 51.39814541379562 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 83.1312672973065 (T) = (0 -11.576094455342181) / Math.Sqrt((1013.8201903383354 / (99)) + (0.7672408622055276 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9585472260987946 = (279.2598266869076 - 11.576094455342181) / 279.2598266869076 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.ReadJson<BinaryData>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromString - Duration of single invocation 49.78 μs 36.05 μs 0.72 0.06 False
DeserializeFromString - Duration of single invocation 50.33 μs 37.06 μs 0.74 0.06 False
DeserializeFromUtf8Bytes - Duration of single invocation 37.83 μs 29.20 μs 0.77 0.09 False
DeserializeFromReader - Duration of single invocation 50.95 μs 34.68 μs 0.68 0.12 False
DeserializeFromUtf8Bytes - Duration of single invocation 37.97 μs 29.31 μs 0.77 0.09 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<BinaryData>.DeserializeFromString(Mode: SourceGen)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 43.84692801276791 (T) = (0 -36546.98799566226) / Math.Sqrt((8394334.948171651 / (89)) + (296223.5134392405 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.3126098112543901 = (53167.747509395434 - 36546.98799566226) / 53167.747509395434 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 49.587274722167926 (T) = (0 -36392.95393745026) / Math.Sqrt((8067775.858289385 / (89)) + (136203.82743855193 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.31447524458667575 = (53087.73118704927 - 36392.95393745026) / 53087.73118704927 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 58.420801653238456 (T) = (0 -29440.239400617287) / Math.Sqrt((1817003.8399309805 / (89)) + (39452.457823321085 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.24586052725371435 = (39038.18917395647 - 29440.239400617287) / 39038.18917395647 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 58.39026656436321 (T) = (0 -35093.9250489069) / Math.Sqrt((6490386.952253018 / (89)) + (185472.46876213085 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.34902184711086587 = (53909.528135706314 - 35093.9250489069) / 53909.528135706314 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 60.39733390870846 (T) = (0 -29532.87241073488) / Math.Sqrt((1807945.7548068622 / (89)) + (32404.586516658506 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.2469577561568962 = (39218.08192328722 - 29532.87241073488) / 39218.08192328722 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.IO.Tests.StringReaderReadLineTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadLine - Duration of single invocation 153.27 μs 85.48 μs 0.56 0.28 False
ReadLine - Duration of single invocation 145.60 μs 77.68 μs 0.53 0.32 False
ReadLine - Duration of single invocation 139.08 μs 87.31 μs 0.63 0.28 False
ReadLineAsync - Duration of single invocation 156.98 μs 97.39 μs 0.62 0.30 False
ReadLineAsync - Duration of single invocation 238.86 μs 192.97 μs 0.81 0.15 False
ReadLineAsync - Duration of single invocation 152.71 μs 90.48 μs 0.59 0.38 False
ReadLineAsync - Duration of single invocation 148.67 μs 99.69 μs 0.67 0.25 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StringReaderReadLineTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StringReaderReadLineTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StringReaderReadLineTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StringReaderReadLineTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 1024])

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 62.562931883053395 (T) = (0 -84404.24498865457) / Math.Sqrt((73575322.39387324 / (108)) + (1318048.6249480965 / (6))) is greater than 1.9813718148759247 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (108) + (6) - 2, .975) and 0.41299209371788653 = (143787.23708039845 - 84404.24498865457) / 143787.23708039845 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 61.98949826427198 (T) = (0 -78768.61126077421) / Math.Sqrt((68392005.59847745 / (109)) + (3230441.5326293847 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.4593842941679928 = (145701.67017169687 - 78768.61126077421) / 145701.67017169687 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 34.49531256291563 (T) = (0 -86714.75011264764) / Math.Sqrt((74335366.62048568 / (109)) + (12361353.840588521 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.3971342808639043 = (143837.58664683995 - 86714.75011264764) / 143837.58664683995 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 22.68994419632432 (T) = (0 -100382.06108713253) / Math.Sqrt((59914884.85432253 / (103)) + (26530724.510870524 / (6))) is greater than 1.9823833701754254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (103) + (6) - 2, .975) and 0.335816338781717 = (151135.99889374894 - 100382.06108713253) / 151135.99889374894 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [  33,  128])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 28.382823372220713 (T) = (0 -193397.2728436303) / Math.Sqrt((283705554.6600324 / (105)) + (335159.7058517498 / (6))) is greater than 1.9819674897361885 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (105) + (6) - 2, .975) and 0.19595936278772513 = (240531.71431007094 - 193397.2728436303) / 240531.71431007094 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 25.15501021968091 (T) = (0 -93145.33052352282) / Math.Sqrt((73831579.89557187 / (109)) + (25074711.322958764 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.3730983291593927 = (148580.44707174733 - 93145.33052352282) / 148580.44707174733 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 73.65258140486661 (T) = (0 -99334.20642329055) / Math.Sqrt((57032858.56562271 / (109)) + (81073.6280171014 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.3520038420542375 = (153294.4373284461 - 99334.20642329055) / 153294.4373284461 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.WriteJson<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeObjectProperty - Duration of single invocation 822.53 μs 695.58 μs 0.85 0.04 False
SerializeToString - Duration of single invocation 818.87 μs 693.38 μs 0.85 0.03 False
SerializeToString - Duration of single invocation 734.66 μs 605.35 μs 0.82 0.03 False
SerializeObjectProperty - Duration of single invocation 737.35 μs 608.23 μs 0.82 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<IndexViewModel>.SerializeObjectProperty(Mode: Reflection)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 49.15357524962136 (T) = (0 -691222.2467403068) / Math.Sqrt((556970957.5519246 / (89)) + (6588114.755461128 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.16168478371254003 = (824537.3975214657 - 691222.2467403068) / 824537.3975214657 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;IndexViewModel&gt;.SerializeToString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 49.02068285275881 (T) = (0 -689214.8734624675) / Math.Sqrt((555760155.5772967 / (89)) + (6594183.056823475 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.16159593389344262 = (822055.738187309 - 689214.8734624675) / 822055.738187309 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;IndexViewModel&gt;.SerializeToString(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 49.16059866034644 (T) = (0 -603474.531279351) / Math.Sqrt((533555719.30607605 / (89)) + (7711216.035004002 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.18019432630493412 = (736118.9982491129 - 603474.531279351) / 736118.9982491129 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;IndexViewModel&gt;.SerializeObjectProperty(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 50.90444826327806 (T) = (0 -605086.0300506452) / Math.Sqrt((542059316.2297032 / (89)) + (5482696.214519198 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.18210424628766247 = (739808.2546635403 - 605086.0300506452) / 739808.2546635403 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in MicroBenchmarks.Serializers.Xml_FromStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 1.48 ms 1.34 ms 0.90 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<IndexViewModel>.XmlSerializer_

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 1/3/2023 12:19:11 AM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 13.402637477231657 (T) = (0 -1352334.2852771506) / Math.Sqrt((2714471957.6261315 / (98)) + (287873987.5079789 / (6))) is greater than 1.9834952585625159 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (98) + (6) - 2, .975) and 0.07937317865670798 = (1468927.750012705 - 1352334.2852771506) / 1468927.750012705 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.WriteJson<StructRecord>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 8.78 μs 7.98 μs 0.91 0.08 False
SerializeToString - Duration of single invocation 12.31 μs 11.02 μs 0.90 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;StructRecord&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;StructRecord&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;StructRecord&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;StructRecord&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 12.248483205769048 (T) = (0 -8000.202796979258) / Math.Sqrt((64987.84861555646 / (16)) + (12098.488906982102 / (6))) is greater than 2.08596344725343 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (6) - 2, .975) and 0.10663420456127384 = (8955.125479200164 - 8000.202796979258) / 8955.125479200164 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;StructRecord&gt;.SerializeToString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 13.920552541678411 (T) = (0 -11076.70915384624) / Math.Sqrt((82056.22128079247 / (16)) + (2607.9490056884347 / (6))) is greater than 2.08596344725343 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (6) - 2, .975) and 0.08570268161700653 = (12114.996873704353 - 11076.70915384624) / 12114.996873704353 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseOrOperatorBenchmark - Duration of single invocation 62.32 ns 11.76 ns 0.19 0.36 False
OnesComplementOperatorBenchmark - Duration of single invocation 43.84 ns 7.67 ns 0.17 0.50 False
OnesComplementBenchmark - Duration of single invocation 44.78 ns 8.94 ns 0.20 0.49 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 79.10 ns 11.60 ns 0.15 0.41 False
BitwiseAndOperatorBenchmark - Duration of single invocation 60.96 ns 11.15 ns 0.18 0.41 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.BitwiseOrOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 39.69754400401307 (T) = (0 -11.229168445177601) / Math.Sqrt((198.929400110319 / (99)) + (0.13317000874050422 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8344061568717509 = (67.81150937164271 - 11.229168445177601) / 67.81150937164271 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 31.529620127066078 (T) = (0 -7.691400171647833) / Math.Sqrt((187.3137706016022 / (99)) + (0.36389143940385715 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8513758249201973 = (51.750666858322276 - 7.691400171647833) / 51.750666858322276 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 33.14307093210957 (T) = (0 -7.738657869571671) / Math.Sqrt((165.18222773198482 / (99)) + (0.36378305195760935 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8492098086696179 = (51.320697992989665 - 7.738657869571671) / 51.320697992989665 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 43.36998528658138 (T) = (0 -11.36374195165921) / Math.Sqrt((166.91606266316944 / (99)) + (0.08120491933765998 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8326495996385785 = (67.90388267442017 - 11.36374195165921) / 67.90388267442017 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 42.60606138168758 (T) = (0 -11.279238305719531) / Math.Sqrt((177.3230131518445 / (99)) + (0.09738964908587336 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8354794216792039 = (68.55822183974041 - 11.279238305719531) / 68.55822183974041 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeObjectProperty - Duration of single invocation 8.91 ms 8.14 ms 0.91 0.02 False
SerializeToString - Duration of single invocation 8.93 ms 8.18 ms 0.92 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;MyEventsListerViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;MyEventsListerViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;MyEventsListerViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;MyEventsListerViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeObjectProperty(Mode: Reflection)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 29.51509143055233 (T) = (0 -8200247.749603175) / Math.Sqrt((29827575122.716137 / (89)) + (1257010597.7133174 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.07748976427434247 = (8889058.822369339 - 8200247.749603175) / 8889058.822369339 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;MyEventsListerViewModel&gt;.SerializeToString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 36.67706413073166 (T) = (0 -8179551.399006918) / Math.Sqrt((28525218636.172653 / (89)) + (222609700.0628954 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.07816675534249085 = (8873135.620147748 - 8179551.399006918) / 8873135.620147748 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.ReadJson<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromString - Duration of single invocation 9.41 μs 8.86 μs 0.94 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<LoginViewModel>.DeserializeFromString(Mode: Reflection)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 20.89826618312018 (T) = (0 -8842.08602091379) / Math.Sqrt((26935.662643534215 / (89)) + (3225.356671226722 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.0641166022492947 = (9447.850065686376 - 8842.08602091379) / 9447.850065686376 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in MicroBenchmarks.Serializers.Xml_FromStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 14.55 ms 13.69 ms 0.94 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;MyEventsListerViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;MyEventsListerViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;MyEventsListerViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;MyEventsListerViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<MyEventsListerViewModel>.XmlSerializer_

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 17.439827298289167 (T) = (0 -13649708.715305606) / Math.Sqrt((136149620927.12424 / (109)) + (2812594191.9211373 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.05029221410156225 = (14372535.339796944 - 13649708.715305606) / 14372535.339796944 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Perf_Convert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToBase64String - Duration of single invocation 41.51 μs 26.72 μs 0.64 0.23 False
ToBase64CharArray - Duration of single invocation 41.78 μs 25.24 μs 0.60 0.22 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Perf_Convert*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Perf_Convert*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Perf_Convert*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Perf_Convert*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Perf_Convert.ToBase64String(formattingOptions: None)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 49.91017368185695 (T) = (0 -26281.370315729146) / Math.Sqrt((5785088.92622285 / (105)) + (206400.37078944332 / (6))) is greater than 1.9819674897361885 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (105) + (6) - 2, .975) and 0.3622953861954628 = (41212.45126161914 - 26281.370315729146) / 41212.45126161914 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Perf_Convert.ToBase64CharArray(binaryDataSize: 1024, formattingOptions: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 48.8994431570108 (T) = (0 -25968.211202137154) / Math.Sqrt((6214845.728932061 / (109)) + (216477.35334822504 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.3648982854884644 = (40888.27129385034 - 25968.211202137154) / 40888.27129385034 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromString - Duration of single invocation 6.08 ms 5.49 ms 0.90 0.02 False
DeserializeFromString - Duration of single invocation 6.03 ms 5.42 ms 0.90 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>.DeserializeFromString(Mode: SourceGen)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 44.82257610723974 (T) = (0 -5524430.188851874) / Math.Sqrt((12867741522.408602 / (89)) + (807497575.9689386 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.11937908717552348 = (6273335.221091884 - 5524430.188851874) / 6273335.221091884 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 31.55396613232889 (T) = (0 -5487877.364045215) / Math.Sqrt((12312439183.719538 / (89)) + (2158889880.5595164 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.11373549664574571 = (6192143.929126338 - 5487877.364045215) / 6192143.929126338 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LastIndexOfAnyValues - Duration of single invocation 1.56 μs 603.97 ns 0.39 0.27 False
IndexOfAnyTwoValues - Duration of single invocation 1.54 μs 589.53 ns 0.38 0.26 False
Reverse - Duration of single invocation 297.54 ns 117.09 ns 0.39 0.29 False
IndexOfAnyThreeValues - Duration of single invocation 2.18 μs 725.34 ns 0.33 0.38 False
SequenceCompareTo - Duration of single invocation 785.14 ns 608.37 ns 0.77 0.16 False
LastIndexOfValue - Duration of single invocation 1.02 μs 505.45 ns 0.49 0.15 False
Reverse - Duration of single invocation 3.59 μs 839.30 ns 0.23 0.39 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span&lt;Byte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.Span&lt;Byte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span&lt;Byte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.Span&lt;Byte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Byte>.LastIndexOfAnyValues(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 52.473520540264026 (T) = (0 -613.6747514300221) / Math.Sqrt((42324.87777524218 / (109)) + (156.694395890694 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.6351281051650057 = (1681.8909872670342 - 613.6747514300221) / 1681.8909872670342 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfAnyTwoValues(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 53.784231277737625 (T) = (0 -603.4896678204731) / Math.Sqrt((41005.05341106911 / (109)) + (181.90685384034063 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.6424598934067911 = (1687.8936284121353 - 603.4896678204731) / 1687.8936284121353 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.Reverse(Size: 33)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 106.43305737846995 (T) = (0 -116.60656604072368) / Math.Sqrt((149.06877092604392 / (79)) + (4.408359539648576 / (6))) is greater than 1.9889597801747605 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (79) + (6) - 2, .975) and 0.5964312149255122 = (288.9385164395242 - 116.60656604072368) / 288.9385164395242 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfAnyThreeValues(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/18/2022 12:43:31 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 55.624439361072575 (T) = (0 -733.4499886572544) / Math.Sqrt((86690.83241440011 / (109)) + (189.57420795903496 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.6856197449345734 = (2333.002715150206 - 733.4499886572544) / 2333.002715150206 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.SequenceCompareTo(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 8.944501690220125 (T) = (0 -647.1040981393747) / Math.Sqrt((666.137564042056 / (107)) + (1046.462590970808 / (6))) is greater than 1.9815667570746824 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (107) + (6) - 2, .975) and 0.15666874670703113 = (767.3190049729773 - 647.1040981393747) / 767.3190049729773 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.LastIndexOfValue(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 38.473629070969146 (T) = (0 -511.26171077596615) / Math.Sqrt((28437.92695393115 / (109)) + (214.1670446133325 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.5644572543931236 = (1173.8496759108784 - 511.26171077596615) / 1173.8496759108784 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.Reverse(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 99.95312301784743 (T) = (0 -844.9155609241571) / Math.Sqrt((64752.97965985276 / (109)) + (193.24078838380606 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.7475062309876288 = (3346.2828181029663 - 844.9155609241571) / 3346.2828181029663 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfAnyFourValues - Duration of single invocation 3.11 μs 1.56 μs 0.50 0.22 False
Reverse - Duration of single invocation 3.57 μs 1.44 μs 0.40 0.18 False
Reverse - Duration of single invocation 292.28 ns 145.02 ns 0.50 0.13 False
LastIndexOfValue - Duration of single invocation 1.06 μs 881.98 ns 0.83 0.18 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.Span&lt;Char&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.Span&lt;Char&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Char>.IndexOfAnyFourValues(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 73.65215480133524 (T) = (0 -1560.4741021384882) / Math.Sqrt((35690.1755466387 / (109)) + (423.17071350888426 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.4849509868001944 = (3029.7584543339867 - 1560.4741021384882) / 3029.7584543339867 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.Reverse(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 103.49578554591804 (T) = (0 -1432.3935280701087) / Math.Sqrt((46891.84848377028 / (109)) + (40.151297209132736 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.601632574878613 = (3595.6592776973225 - 1432.3935280701087) / 3595.6592776973225 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.Reverse(Size: 33)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 99.02763960947065 (T) = (0 -145.5179088825622) / Math.Sqrt((148.7060946663345 / (79)) + (2.279827640634437 / (6))) is greater than 1.9889597801747605 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (79) + (6) - 2, .975) and 0.5058239311361314 = (294.46571384388153 - 145.5179088825622) / 294.46571384388153 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.LastIndexOfValue(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/23/2022 1:02:40 AM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 16.315582642471185 (T) = (0 -902.8094132888242) / Math.Sqrt((3442.6372199242983 / (109)) + (536.3085218640308 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.16580906446323948 = (1082.2575202257635 - 902.8094132888242) / 1082.2575202257635 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in MicroBenchmarks.Serializers.Json_ToString<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 8.29 ms 7.52 ms 0.91 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;MyEventsListerViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;MyEventsListerViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;MyEventsListerViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;MyEventsListerViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_ToString<MyEventsListerViewModel>.SystemTextJson_SourceGen_

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 34.53501381130639 (T) = (0 -7542913.299303647) / Math.Sqrt((32328445396.7527 / (89)) + (320570811.5505481 / (6))) is greater than 1.9858018143453997 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (89) + (6) - 2, .975) and 0.0854701060170028 = (8247858.652769074 - 7542913.299303647) / 8247858.652769074 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
MatchesWord - Duration of single invocation 50.74 μs 35.86 μs 0.71 0.07 False
MatchesWord - Duration of single invocation 50.72 μs 36.31 μs 0.72 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: Compiled)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 38.85209198655266 (T) = (0 -36066.14275697036) / Math.Sqrt((20446815.906486187 / (109)) + (98749.52560590424 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.3273262368185161 = (53616.098517640414 - 36066.14275697036) / 53616.098517640414 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 39.32506658554302 (T) = (0 -36066.67300870724) / Math.Sqrt((20136381.05947617 / (109)) + (104125.43665713802 / (6))) is greater than 1.9811803594144102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (109) + (6) - 2, .975) and 0.3289309864957909 = (53745.10263910587 - 36066.67300870724) / 53745.10263910587 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AddOperatorBenchmark - Duration of single invocation 172.58 ns 10.98 ns 0.06 0.29 False
BitwiseAndBenchmark - Duration of single invocation 62.27 ns 11.07 ns 0.18 0.29 False
BitwiseAndOperatorBenchmark - Duration of single invocation 63.69 ns 12.18 ns 0.19 0.32 False
EqualsBenchmark - Duration of single invocation 59.43 ns 10.51 ns 0.18 0.26 False
EqualsStaticBenchmark - Duration of single invocation 185.66 ns 10.93 ns 0.06 0.21 False
ConditionalSelectBenchmark - Duration of single invocation 91.76 ns 14.79 ns 0.16 0.24 False
BitwiseOrOperatorBenchmark - Duration of single invocation 63.88 ns 11.15 ns 0.17 0.36 False
EqualsAllBenchmark - Duration of single invocation 55.12 ns 9.21 ns 0.17 0.37 False
BitwiseOrBenchmark - Duration of single invocation 63.87 ns 11.14 ns 0.17 0.34 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.AddOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 82.94740752423233 (T) = (0 -11.414454795605552) / Math.Sqrt((397.3605391937192 / (99)) + (0.7252745197888026 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9366137602695752 = (180.07780307129863 - 11.414454795605552) / 180.07780307129863 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 42.76557917587153 (T) = (0 -11.094813860573515) / Math.Sqrt((175.64218886212643 / (99)) + (0.04422166720067668 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8372616751763604 = (68.17579001502583 - 11.094813860573515) / 68.17579001502583 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 41.77612402348196 (T) = (0 -11.33436552065776) / Math.Sqrt((177.28567353517207 / (99)) + (0.19576862205980716 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8326930666830562 = (67.74594032625122 - 11.33436552065776) / 67.74594032625122 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 89.52965006017986 (T) = (0 -10.511623179193725) / Math.Sqrt((36.7941377766178 / (99)) + (0.013715285423324106 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8389266482175165 = (65.25985250116868 - 10.511623179193725) / 65.25985250116868 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.EqualsStaticBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 73.3924442167445 (T) = (0 -10.964510853196062) / Math.Sqrt((609.2042628507218 / (99)) + (0.07434189890528219 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9432502219296338 = (193.20799527358045 - 10.964510853196062) / 193.20799527358045 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 68.91219609606448 (T) = (0 -14.76367271304791) / Math.Sqrt((113.62756057253807 / (90)) + (0.009375222577038404 / (6))) is greater than 1.98552344186624 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (90) + (6) - 2, .975) and 0.839947861905577 = (92.2428959014472 - 14.76367271304791) / 92.2428959014472 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 37.814425182401095 (T) = (0 -11.69074691430225) / Math.Sqrt((189.3528253420698 / (99)) + (1.7116260659654468 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.827448343809083 = (67.75215707792002 - 11.69074691430225) / 67.75215707792002 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 94.12014421891767 (T) = (0 -9.28961423275552) / Math.Sqrt((23.80588857826779 / (99)) + (0.007912155740945627 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8328296989014111 = (55.56976431643175 - 9.28961423275552) / 55.56976431643175 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 50.87898308449137 (T) = (0 -11.04952682269107) / Math.Sqrt((105.21613927208658 / (92)) + (0.010442980189339705 / (6))) is greater than 1.9849843115220658 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (92) + (6) - 2, .975) and 0.8313095357829118 = (65.50178680207678 - 11.04952682269107) / 65.50178680207678 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2b74314b3c049f5112e526a27a8cac74f2a62d43
Compare ddee0f24992fcb8450885e787c21a82931af3b89
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EqualsStaticBenchmark - Duration of single invocation 185.62 ns 11.31 ns 0.06 0.35 False
BitwiseOrOperatorBenchmark - Duration of single invocation 62.22 ns 10.82 ns 0.17 0.32 False
BitwiseAndBenchmark - Duration of single invocation 61.99 ns 10.88 ns 0.18 0.34 False
AddOperatorBenchmark - Duration of single invocation 175.92 ns 11.10 ns 0.06 0.21 False
BitwiseOrBenchmark - Duration of single invocation 62.44 ns 11.43 ns 0.18 0.32 False
BitwiseAndOperatorBenchmark - Duration of single invocation 62.22 ns 11.92 ns 0.19 0.37 False
EqualsBenchmark - Duration of single invocation 53.77 ns 11.17 ns 0.21 0.36 False
EqualsAllBenchmark - Duration of single invocation 53.96 ns 10.14 ns 0.19 0.43 False
ConditionalSelectBenchmark - Duration of single invocation 91.22 ns 15.36 ns 0.17 0.24 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.EqualsStaticBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 74.61039326893663 (T) = (0 -11.333075536066934) / Math.Sqrt((574.4673060714365 / (99)) + (0.15407157559755674 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9408063891871482 = (191.4577499233475 - 11.333075536066934) / 191.4577499233475 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 42.391879614829435 (T) = (0 -11.163289541423808) / Math.Sqrt((172.64404532879115 / (99)) + (0.07540480844259294 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.834238949906901 = (67.34567339645825 - 11.163289541423808) / 67.34567339645825 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 40.92261768878764 (T) = (0 -11.049325293365385) / Math.Sqrt((191.17989312191065 / (99)) + (0.057311815870895046 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8376475882770934 = (68.05766034583897 - 11.049325293365385) / 68.05766034583897 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 89.76116039209899 (T) = (0 -11.149768749906825) / Math.Sqrt((351.4812681607267 / (99)) + (0.10741049873879298 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.9382988950154277 = (180.70614379912178 - 11.149768749906825) / 180.70614379912178 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 43.834733774517 (T) = (0 -11.770414717704703) / Math.Sqrt((130.31184773789786 / (92)) + (0.665477765000259 / (6))) is greater than 1.9849843115220658 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (92) + (6) - 2, .975) and 0.8215087750564969 = (65.94394050144669 - 11.770414717704703) / 65.94394050144669 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 40.30480992832204 (T) = (0 -11.886206435225413) / Math.Sqrt((179.50049031628058 / (99)) + (0.7166811568731389 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.824988963011408 = (67.91689621266688 - 11.886206435225413) / 67.91689621266688 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 93.39939604683546 (T) = (0 -10.835827331591574) / Math.Sqrt((31.940238795593046 / (99)) + (0.12565522414474234 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8347739525298455 = (65.58183468952676 - 10.835827331591574) / 65.58183468952676 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 65.71684033062944 (T) = (0 -10.385412369215912) / Math.Sqrt((34.84120449707012 / (99)) + (0.6966821658154225 / (6))) is greater than 1.983264144773014 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (99) + (6) - 2, .975) and 0.8123507519613404 = (55.34481207767114 - 10.385412369215912) / 55.34481207767114 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM falls between 2/27/2023 12:59:46 AM and 3/8/2023 2:17:54 AM.
IsImprovementStdDev: Marked as improvement because 67.05116030258131 (T) = (0 -14.924324310000314) / Math.Sqrt((115.25916001005082 / (90)) + (0.1835025724456283 / (6))) is greater than 1.98552344186624 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (90) + (6) - 2, .975) and 0.837255878694457 = (91.70422986880565 - 14.924324310000314) / 91.70422986880565 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kotlarmilos
Copy link
Member

Most likely dotnet/runtime#81782.

/cc: @BrzVlad @SamMonoRT

@kotlarmilos kotlarmilos mentioned this issue Apr 4, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-improvement runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kotlarmilos