Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Changes at 5/11/2022 9:09:01 PM #69444

Closed
performanceautofiler bot opened this issue May 17, 2022 · 8 comments
Closed

[Perf] Changes at 5/11/2022 9:09:01 PM #69444

performanceautofiler bot opened this issue May 17, 2022 · 8 comments
Assignees
@kunalspathak
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime
Milestone
7.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in Benchstone.BenchI.HeapSort

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 215.09 μs 238.04 μs 1.11 0.00 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.BenchI.HeapSort*'

Payloads

Baseline
Compare

Histogram

Benchstone.BenchI.HeapSort.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 238.04431299751244 > 226.9287108834691.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -66.23321692467101 (T) = (0 -236664.2136185022) / Math.Sqrt((1213106.0727915505 / (37)) + (2089608.7579848508 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.09719565720268247 = (215699.1891691235 - 236664.2136185022) / 215699.1891691235 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 2.07 ms 2.26 ms 1.10 0.08 False
Count - Duration of single invocation 5.01 ms 5.35 ms 1.07 0.00 False
Count - Duration of single invocation 2.50 ms 2.72 ms 1.09 0.06 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Huck[a-zA-Z]+|Saw[a-zA-Z]+", Options: Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.2648206988324175 > 2.1817654359375003.
IsChangePoint: Marked as a change because one of 5/5/2022 1:04:22 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -17.545676213916142 (T) = (0 -2260787.409644609) / Math.Sqrt((2072386870.7561069 / (37)) + (912917384.0286242 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.07728993009671208 = (2098587.7120764046 - 2260787.409644609) / 2098587.7120764046 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Twain", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.353628029647436 > 5.247663791141827.
IsChangePoint: Marked as a change because one of 3/21/2022 1:28:14 PM, 5/5/2022 1:04:22 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -37.62417890964721 (T) = (0 -5358651.178812367) / Math.Sqrt((3468110800.2975383 / (37)) + (374664673.6606075 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.07784853653979781 = (4971617.993762992 - 5358651.178812367) / 4971617.993762992 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Huck[a-zA-Z]+|Saw[a-zA-Z]+", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.71624684375 > 2.6176171298883935.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -16.561258509398872 (T) = (0 -2667867.2814934123) / Math.Sqrt((2183552242.712746 / (37)) + (1060727166.983088 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.06376106230900554 = (2507957.2622281597 - 2667867.2814934123) / 2507957.2622281597 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in Benchstone.BenchI.IniArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 51.11 ms 102.12 ms 2.00 0.07 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.BenchI.IniArray*'

Payloads

Baseline
Compare

Histogram

Benchstone.BenchI.IniArray.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 102.121077125 > 53.434593568125.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -109.7761885454756 (T) = (0 -103462588.3596774) / Math.Sqrt((8289312031551.029 / (37)) + (564033315407.9225 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -1.0929363425294765 = (49434178.31554055 - 103462588.3596774) / 49434178.31554055 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 1.61 secs 1.71 secs 1.06 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.710382038 > 1.6901519082.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/6/2022 2:58:43 AM, 5/5/2022 1:04:22 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -42.71277925239627 (T) = (0 -1699637154.3143096) / Math.Sqrt((106703282340818.9 / (37)) + (73320280279635.73 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.06109354790888636 = (1601778804.2004507 - 1699637154.3143096) / 1601778804.2004507 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in Span.Sorting

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BubbleSortSpan - Duration of single invocation 199.99 μs 336.17 μs 1.68 0.03 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Span.Sorting*'

Payloads

Baseline
Compare

Histogram

Span.Sorting.BubbleSortSpan(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 336.17043585714276 > 207.19591918846154.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -18.6756977544105 (T) = (0 -320411.8755953267) / Math.Sqrt((3879716.9322020453 / (37)) + (1331861623.1531248 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.619467448905977 = (197850.14870893472 - 320411.8755953267) / 197850.14870893472 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArrayBoolArrayCtor - Duration of single invocation 29.91 ns 33.39 ns 1.12 0.09 False
BitArrayCopyToBoolArray - Duration of single invocation 115.49 ns 126.43 ns 1.09 0.05 False
BitArraySetLengthShrink - Duration of single invocation 14.01 ns 15.19 ns 1.08 0.17 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Tests.Perf_BitArray*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_BitArray.BitArrayBoolArrayCtor(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 33.38690186770933 > 31.919501309978056.
IsChangePoint: Marked as a change because one of 3/21/2022 1:28:14 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -24.13824770446625 (T) = (0 -34.61496284733508) / Math.Sqrt((0.4622537262069015 / (37)) + (0.6713160296965065 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.14792534883509728 = (30.15436751393458 - 34.61496284733508) / 30.15436751393458 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayCopyToBoolArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 126.4318613974565 > 120.44977664309093.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/8/2022 7:02:49 PM, 4/21/2022 3:06:05 PM, 5/11/2022 3:41:15 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -11.55122615151952 (T) = (0 -122.43554735204845) / Math.Sqrt((2.464098937401943 / (36)) + (12.291944954613985 / (32))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (32) - 2, .025) and -0.06777064922601404 = (114.66464960504138 - 122.43554735204845) / 114.66464960504138 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArraySetLengthShrink(Size: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.194876014433705 > 14.834819748246597.
IsChangePoint: Marked as a change because one of 3/31/2022 9:26:55 AM, 5/13/2022 11:25:51 AM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -11.445201690372365 (T) = (0 -15.249377312943109) / Math.Sqrt((0.17765887248015655 / (50)) + (0.04587436347345796 / (18))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (18) - 2, .025) and -0.06227833343904642 = (14.355350036721914 - 15.249377312943109) / 14.355350036721914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfString - Duration of single invocation 22.46 ns 36.03 ns 1.60 0.05 True
IndexOfString - Duration of single invocation 14.28 ns 19.88 ns 1.39 0.03 False
IndexOfString - Duration of single invocation 22.42 ns 36.13 ns 1.61 0.04 True

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだxだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ", value: "x", comparisonType: Ordinal)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.03105245233972 > 23.555543254782663.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -197.8239852882902 (T) = (0 -35.581321397017604) / Math.Sqrt((0.0195347284217183 / (37)) + (0.11876754470666598 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.5799751163319484 = (22.520178342822753 - 35.581321397017604) / 22.520178342822753 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "~", comparisonType: Ordinal)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.877352581665676 > 14.962572289919992.
IsChangePoint: Marked as a change because one of 3/14/2022 4:00:26 PM, 4/8/2022 10:19:59 AM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -119.90119436238683 (T) = (0 -19.659969463437104) / Math.Sqrt((0.05071977257029071 / (37)) + (0.01651890530728478 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.36257426966277934 = (14.428548888056362 - 19.659969463437104) / 14.428548888056362 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "X", comparisonType: Ordinal)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.12665293769705 > 23.551572384849607.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -214.2963290281721 (T) = (0 -35.58508805765246) / Math.Sqrt((0.021872398984698553 / (37)) + (0.09707861032679886 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.5808567498014523 = (22.510001657089912 - 35.58508805765246) / 22.510001657089912 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 78.78 μs 89.81 μs 1.14 0.02 False
Count - Duration of single invocation 38.19 μs 50.60 μs 1.32 0.37 True
Count - Duration of single invocation 42.63 μs 56.34 μs 1.32 0.26 True
Count - Duration of single invocation 77.43 μs 87.29 μs 1.13 0.01 False
Count - Duration of single invocation 60.46 μs 72.02 μs 1.19 0.12 False
Count - Duration of single invocation 91.54 μs 101.87 μs 1.11 0.03 False
Count - Duration of single invocation 75.85 μs 88.41 μs 1.17 0.01 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: NonBacktracking)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 89.81193195585426 > 81.7740405031082.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -42.81879052980991 (T) = (0 -89718.09370032944) / Math.Sqrt((2315107.5226963223 / (37)) + (322433.3994665335 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.14800442863488011 = (78151.34808061272 - 89718.09370032944) / 78151.34808061272 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 50.60256728213129 > 40.09436088887635.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -60.082669865084675 (T) = (0 -50532.174842069406) / Math.Sqrt((1455577.3141548173 / (37)) + (23239.230746782025 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.31244926755805386 = (38502.19287796906 - 50532.174842069406) / 38502.19287796906 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 56.34297109455293 > 47.19790196752282.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -53.539905929519676 (T) = (0 -56498.61343298952) / Math.Sqrt((1817980.461379894 / (37)) + (20154.733442785317 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.26813500340931756 = (44552.5226266098 - 56498.61343298952) / 44552.5226266098 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 87.289007278685 > 81.65243312418319.
IsChangePoint: Marked as a change because one of 4/13/2022 5:30:39 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -60.59497957926352 (T) = (0 -87065.22547575379) / Math.Sqrt((409040.07872280787 / (37)) + (400047.0443363079 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.12073624250266345 = (77685.74101015287 - 87065.22547575379) / 77685.74101015287 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 72.02221843317972 > 63.22909917428216.
IsChangePoint: Marked as a change because one of 4/6/2022 11:11:30 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -42.54586044524999 (T) = (0 -71815.0535166913) / Math.Sqrt((2025730.8379846802 / (37)) + (385044.8524190948 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.1813951139523011 = (60788.34478706913 - 71815.0535166913) / 60788.34478706913 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 101.86718141233767 > 95.54391778758875.
IsChangePoint: Marked as a change because one of 3/18/2022 7:38:50 AM, 4/6/2022 1:01:14 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -35.48605983171002 (T) = (0 -102484.75637328616) / Math.Sqrt((2365733.44482041 / (37)) + (823612.661316888 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.11628222787677776 = (91809.00117725342 - 102484.75637328616) / 91809.00117725342 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 88.41445896388217 > 79.70534152680308.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -52.114040337106 (T) = (0 -87778.56423858648) / Math.Sqrt((1551252.8914258825 / (37)) + (292123.1711855977 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.1554471231627319 = (75969.3476914078 - 87778.56423858648) / 75969.3476914078 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in BenchmarksGame.KNucleotide_9

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
RunBench - Duration of single invocation 64.77 ms 70.94 ms 1.10 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'BenchmarksGame.KNucleotide_9*'

Payloads

Baseline
Compare

Histogram

BenchmarksGame.KNucleotide_9.RunBench

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 70.94013678947368 > 67.89339493781249.
IsChangePoint: Marked as a change because one of 3/11/2022 8:42:43 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -32.15404641575599 (T) = (0 -71078554.73129775) / Math.Sqrt((759929468162.4705 / (37)) + (735822072376.3405 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.10520016977791523 = (64312833.706477486 - 71078554.73129775) / 64312833.706477486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<ClassImplementingIXmlSerialiable>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 467.97 ns 514.84 ns 1.10 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;ClassImplementingIXmlSerialiable&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<ClassImplementingIXmlSerialiable>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 514.8427245439227 > 492.6313153801929.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -21.11335698885973 (T) = (0 -509.21851498778585) / Math.Sqrt((58.552437282029196 / (37)) + (46.26253048116442 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.07840545910621191 = (472.1957874822229 - 509.21851498778585) / 472.1957874822229 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in BenchmarksGame.FannkuchRedux_2

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
RunBench - Duration of single invocation 120.69 ms 145.90 ms 1.21 0.01 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'BenchmarksGame.FannkuchRedux_2*'

Payloads

Baseline
Compare

Histogram

BenchmarksGame.FannkuchRedux_2.RunBench(n: 10, expectedSum: 73196)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 145.9013087222222 > 129.20856829125.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -61.857940604691365 (T) = (0 -145473490.63470155) / Math.Sqrt((2331835521030.1094 / (37)) + (2341886474517.9634 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.18805173159588984 = (122447101.22116439 - 145473490.63470155) / 122447101.22116439 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in ByteMark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BenchLUDecomp - Duration of single invocation 932.06 ms 1.01 secs 1.08 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'ByteMark*'

Payloads

Baseline
Compare

Histogram

ByteMark.BenchLUDecomp

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0059008590666667 > 977.9626353699999.
IsChangePoint: Marked as a change because one of 3/10/2022 7:11:56 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -43.12792726346629 (T) = (0 -1003908583.6688172) / Math.Sqrt((73292221578591.89 / (37)) + (17788590398797.06 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.07372725046436017 = (934975416.9270192 - 1003908583.6688172) / 934975416.9270192 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Ctor_ByteArray - Duration of single invocation 103.01 ns 134.24 ns 1.30 0.01 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_BigInteger.Ctor_ByteArray(numberString: 12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 134.2382737558327 > 107.79874023595556.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -121.06568622478281 (T) = (0 -133.1602774229258) / Math.Sqrt((0.9755310705152499 / (37)) + (1.1906071454690825 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.3010470078747487 = (102.3485520638045 - 133.1602774229258) / 102.3485520638045 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 468.07 μs 539.87 μs 1.15 0.17 False
Count - Duration of single invocation 5.34 ms 6.48 ms 1.21 0.00 True
Count - Duration of single invocation 545.44 μs 588.33 μs 1.08 0.12 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w.+-]+@[\w.-]+.[\w.-]+", Options: Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 539.8737181697612 > 505.0610244906339.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -20.39339277462509 (T) = (0 -536389.3334464954) / Math.Sqrt((125593746.03992862 / (37)) + (118103224.07039005 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.11364467378637334 = (481652.1338200097 - 536389.3334464954) / 481652.1338200097 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.477677295673077 > 5.61145696784856.
IsChangePoint: Marked as a change because one of 5/11/2022 9:09:01 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -213.2205046301869 (T) = (0 -6497187.545453805) / Math.Sqrt((487774854.6559917 / (37)) + (490151730.0971527 / (30))) is less than -1.997137908391408 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (30) - 2, .025) and -0.21700217871719132 = (5338681.934244614 - 6497187.545453805) / 5338681.934244614 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w\.+-]+@[\w\.-]+\.[\w\.-]+", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 588.328496031746 > 573.3062299299569.
IsChangePoint: Marked as a change because one of 4/3/2022 3:34:31 AM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -18.46640693838835 (T) = (0 -587738.8570414352) / Math.Sqrt((165165976.94906864 / (37)) + (26432337.566885512 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.07810447081885356 = (545159.4654783589 - 587738.8570414352) / 545159.4654783589 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 1.46 μs 1.70 μs 1.16 0.02 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;Location&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<Location>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.6953511534087833 > 1.5426981342391854.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -57.4347245965593 (T) = (0 -1713.3533787693557) / Math.Sqrt((364.07619111733334 / (37)) + (287.44130211068057 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.17171345317224157 = (1462.2631276705954 - 1713.3533787693557) / 1462.2631276705954 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in System.Net.Primitives.Tests.IPAddressPerformanceTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryWriteBytes - Duration of single invocation 13.37 ns 15.93 ns 1.19 0.00 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*'

Payloads

Baseline
Compare

Histogram

System.Net.Primitives.Tests.IPAddressPerformanceTests.TryWriteBytes(address: 1020:3040:5060:7080:9010:1112:1314:1516)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.93269249351485 > 14.016778858798101.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -70.74900854339784 (T) = (0 -15.986957158464076) / Math.Sqrt((0.0039061577740166498 / (37)) + (0.03810768129290934 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.19286974607202606 = (13.402097933246415 - 15.986957158464076) / 13.402097933246415 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 556.19 μs 596.06 μs 1.07 0.02 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<MyEventsListerViewModel>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 596.0583336500396 > 573.0893492699146.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -34.158508429466664 (T) = (0 -608871.2171722453) / Math.Sqrt((72543954.04460804 / (37)) + (54206686.29842921 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.12113649633594999 = (543083.9323865845 - 608871.2171722453) / 543083.9323865845 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 51.21 μs 59.81 μs 1.17 0.03 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<IndexViewModel>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 59.80912206572769 > 53.28713376979103.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -37.84884227959595 (T) = (0 -60023.50566304724) / Math.Sqrt((1109804.728581303 / (37)) + (858366.5740530189 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.17847586456300038 = (50933.165004024086 - 60023.50566304724) / 50933.165004024086 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in Benchstone.BenchI.CSieve

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 3.49 ms 4.38 ms 1.25 0.00 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.BenchI.CSieve*'

Payloads

Baseline
Compare

Histogram

Benchstone.BenchI.CSieve.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.3768314598958336 > 3.69367831.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -114.80472880432049 (T) = (0 -4332892.052626322) / Math.Sqrt((732356384.2350198 / (37)) + (943355311.8057752 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.23118645795326329 = (3519281.766491621 - 4332892.052626322) / 3519281.766491621 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in System.IO.Tests.StringReaderReadLineTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadLineAsync - Duration of single invocation 2.87 μs 3.05 μs 1.06 0.01 False
ReadLine - Duration of single invocation 2.63 μs 2.88 μs 1.09 0.00 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.StringReaderReadLineTests*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [1025, 2048])

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.053503130279402 > 3.032082424200024.
IsChangePoint: Marked as a change because one of 3/22/2022 2:38:09 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -27.72143008752571 (T) = (0 -3075.192109721527) / Math.Sqrt((1322.2042940928104 / (37)) + (584.6746382404815 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.0713605770165008 = (2870.3614597106493 - 3075.192109721527) / 2870.3614597106493 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.8753147528093317 > 2.7979686910339416.
IsChangePoint: Marked as a change because one of 3/14/2022 2:55:27 AM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -32.86582434592007 (T) = (0 -2893.8674635833995) / Math.Sqrt((972.6306406437185 / (37)) + (748.0810629472584 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.08771589899545863 = (2660.4993696019164 - 2893.8674635833995) / 2660.4993696019164 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 837.11 ns 967.85 ns 1.16 0.03 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<LoginViewModel>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 967.852390828133 > 881.3995054092784.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -35.970822412823246 (T) = (0 -970.3842080919786) / Math.Sqrt((314.5531193853344 / (37)) + (119.59236324317129 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.14984492776010191 = (843.9261544443973 - 970.3842080919786) / 843.9261544443973 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Indexer - Duration of single invocation 6.69 ns 12.75 ns 1.90 0.02 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*'

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Primitives.StringSegmentBenchmark.Indexer

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.745469787365078 > 7.01748748907736.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -268.0872480834625 (T) = (0 -12.822662932333175) / Math.Sqrt((0.007576291427929072 / (37)) + (0.009519011308431346 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.8975310597339127 = (6.757551011645245 - 12.822662932333175) / 6.757551011645245 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added CoreClr untriaged New issue has not been triaged by the area owner labels May 17, 2022
@DrewScoggins DrewScoggins assigned EgorBo and unassigned dakersnar May 17, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@DrewScoggins DrewScoggins transferred this issue from dotnet/perf-autofiling-issues May 17, 2022
@DrewScoggins DrewScoggins assigned kunalspathak and unassigned EgorBo May 17, 2022
@DrewScoggins
Copy link
Member

This range seems to be implicated 336a5d2...9d972ed

@DrewScoggins
Copy link
Member

Also from Windows-Amd64 dotnet/perf-autofiling-issues#5354

@jeffschwMSFT jeffschwMSFT added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label May 18, 2022
@ghost
Copy link

ghost commented May 18, 2022

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in Benchstone.BenchI.HeapSort

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 215.09 μs 238.04 μs 1.11 0.00 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.BenchI.HeapSort*'

Payloads

Baseline
Compare

Histogram

Benchstone.BenchI.HeapSort.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 238.04431299751244 > 226.9287108834691.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -66.23321692467101 (T) = (0 -236664.2136185022) / Math.Sqrt((1213106.0727915505 / (37)) + (2089608.7579848508 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.09719565720268247 = (215699.1891691235 - 236664.2136185022) / 215699.1891691235 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 2.07 ms 2.26 ms 1.10 0.08 False
Count - Duration of single invocation 5.01 ms 5.35 ms 1.07 0.00 False
Count - Duration of single invocation 2.50 ms 2.72 ms 1.09 0.06 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Huck[a-zA-Z]+|Saw[a-zA-Z]+", Options: Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.2648206988324175 > 2.1817654359375003.
IsChangePoint: Marked as a change because one of 5/5/2022 1:04:22 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -17.545676213916142 (T) = (0 -2260787.409644609) / Math.Sqrt((2072386870.7561069 / (37)) + (912917384.0286242 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.07728993009671208 = (2098587.7120764046 - 2260787.409644609) / 2098587.7120764046 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Twain", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.353628029647436 > 5.247663791141827.
IsChangePoint: Marked as a change because one of 3/21/2022 1:28:14 PM, 5/5/2022 1:04:22 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -37.62417890964721 (T) = (0 -5358651.178812367) / Math.Sqrt((3468110800.2975383 / (37)) + (374664673.6606075 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.07784853653979781 = (4971617.993762992 - 5358651.178812367) / 4971617.993762992 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Huck[a-zA-Z]+|Saw[a-zA-Z]+", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.71624684375 > 2.6176171298883935.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -16.561258509398872 (T) = (0 -2667867.2814934123) / Math.Sqrt((2183552242.712746 / (37)) + (1060727166.983088 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.06376106230900554 = (2507957.2622281597 - 2667867.2814934123) / 2507957.2622281597 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in Benchstone.BenchI.IniArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 51.11 ms 102.12 ms 2.00 0.07 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.BenchI.IniArray*'

Payloads

Baseline
Compare

Histogram

Benchstone.BenchI.IniArray.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 102.121077125 > 53.434593568125.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -109.7761885454756 (T) = (0 -103462588.3596774) / Math.Sqrt((8289312031551.029 / (37)) + (564033315407.9225 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -1.0929363425294765 = (49434178.31554055 - 103462588.3596774) / 49434178.31554055 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 1.61 secs 1.71 secs 1.06 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.710382038 > 1.6901519082.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/6/2022 2:58:43 AM, 5/5/2022 1:04:22 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -42.71277925239627 (T) = (0 -1699637154.3143096) / Math.Sqrt((106703282340818.9 / (37)) + (73320280279635.73 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.06109354790888636 = (1601778804.2004507 - 1699637154.3143096) / 1601778804.2004507 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in Span.Sorting

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BubbleSortSpan - Duration of single invocation 199.99 μs 336.17 μs 1.68 0.03 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Span.Sorting*'

Payloads

Baseline
Compare

Histogram

Span.Sorting.BubbleSortSpan(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 336.17043585714276 > 207.19591918846154.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -18.6756977544105 (T) = (0 -320411.8755953267) / Math.Sqrt((3879716.9322020453 / (37)) + (1331861623.1531248 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.619467448905977 = (197850.14870893472 - 320411.8755953267) / 197850.14870893472 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArrayBoolArrayCtor - Duration of single invocation 29.91 ns 33.39 ns 1.12 0.09 False
BitArrayCopyToBoolArray - Duration of single invocation 115.49 ns 126.43 ns 1.09 0.05 False
BitArraySetLengthShrink - Duration of single invocation 14.01 ns 15.19 ns 1.08 0.17 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Tests.Perf_BitArray*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_BitArray.BitArrayBoolArrayCtor(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 33.38690186770933 > 31.919501309978056.
IsChangePoint: Marked as a change because one of 3/21/2022 1:28:14 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -24.13824770446625 (T) = (0 -34.61496284733508) / Math.Sqrt((0.4622537262069015 / (37)) + (0.6713160296965065 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.14792534883509728 = (30.15436751393458 - 34.61496284733508) / 30.15436751393458 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayCopyToBoolArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 126.4318613974565 > 120.44977664309093.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/8/2022 7:02:49 PM, 4/21/2022 3:06:05 PM, 5/11/2022 3:41:15 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -11.55122615151952 (T) = (0 -122.43554735204845) / Math.Sqrt((2.464098937401943 / (36)) + (12.291944954613985 / (32))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (32) - 2, .025) and -0.06777064922601404 = (114.66464960504138 - 122.43554735204845) / 114.66464960504138 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArraySetLengthShrink(Size: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.194876014433705 > 14.834819748246597.
IsChangePoint: Marked as a change because one of 3/31/2022 9:26:55 AM, 5/13/2022 11:25:51 AM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -11.445201690372365 (T) = (0 -15.249377312943109) / Math.Sqrt((0.17765887248015655 / (50)) + (0.04587436347345796 / (18))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (18) - 2, .025) and -0.06227833343904642 = (14.355350036721914 - 15.249377312943109) / 14.355350036721914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfString - Duration of single invocation 22.46 ns 36.03 ns 1.60 0.05 True
IndexOfString - Duration of single invocation 14.28 ns 19.88 ns 1.39 0.03 False
IndexOfString - Duration of single invocation 22.42 ns 36.13 ns 1.61 0.04 True

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだxだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ", value: "x", comparisonType: Ordinal)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.03105245233972 > 23.555543254782663.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -197.8239852882902 (T) = (0 -35.581321397017604) / Math.Sqrt((0.0195347284217183 / (37)) + (0.11876754470666598 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.5799751163319484 = (22.520178342822753 - 35.581321397017604) / 22.520178342822753 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "~", comparisonType: Ordinal)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.877352581665676 > 14.962572289919992.
IsChangePoint: Marked as a change because one of 3/14/2022 4:00:26 PM, 4/8/2022 10:19:59 AM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -119.90119436238683 (T) = (0 -19.659969463437104) / Math.Sqrt((0.05071977257029071 / (37)) + (0.01651890530728478 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.36257426966277934 = (14.428548888056362 - 19.659969463437104) / 14.428548888056362 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "X", comparisonType: Ordinal)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.12665293769705 > 23.551572384849607.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -214.2963290281721 (T) = (0 -35.58508805765246) / Math.Sqrt((0.021872398984698553 / (37)) + (0.09707861032679886 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.5808567498014523 = (22.510001657089912 - 35.58508805765246) / 22.510001657089912 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 130ef3cae8b88593cbd1627d31c2c4bbf6336c40
Compare 7b5f40f08ca09b510151c0c3bcdeec12c61d6427
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 78.78 μs 89.81 μs 1.14 0.02 False
Count - Duration of single invocation 38.19 μs 50.60 μs 1.32 0.37 True
Count - Duration of single invocation 42.63 μs 56.34 μs 1.32 0.26 True
Count - Duration of single invocation 77.43 μs 87.29 μs 1.13 0.01 False
Count - Duration of single invocation 60.46 μs 72.02 μs 1.19 0.12 False
Count - Duration of single invocation 91.54 μs 101.87 μs 1.11 0.03 False
Count - Duration of single invocation 75.85 μs 88.41 μs 1.17 0.01 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: NonBacktracking)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 89.81193195585426 > 81.7740405031082.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -42.81879052980991 (T) = (0 -89718.09370032944) / Math.Sqrt((2315107.5226963223 / (37)) + (322433.3994665335 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.14800442863488011 = (78151.34808061272 - 89718.09370032944) / 78151.34808061272 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 50.60256728213129 > 40.09436088887635.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -60.082669865084675 (T) = (0 -50532.174842069406) / Math.Sqrt((1455577.3141548173 / (37)) + (23239.230746782025 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.31244926755805386 = (38502.19287796906 - 50532.174842069406) / 38502.19287796906 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 56.34297109455293 > 47.19790196752282.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -53.539905929519676 (T) = (0 -56498.61343298952) / Math.Sqrt((1817980.461379894 / (37)) + (20154.733442785317 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.26813500340931756 = (44552.5226266098 - 56498.61343298952) / 44552.5226266098 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 87.289007278685 > 81.65243312418319.
IsChangePoint: Marked as a change because one of 4/13/2022 5:30:39 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -60.59497957926352 (T) = (0 -87065.22547575379) / Math.Sqrt((409040.07872280787 / (37)) + (400047.0443363079 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.12073624250266345 = (77685.74101015287 - 87065.22547575379) / 77685.74101015287 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 72.02221843317972 > 63.22909917428216.
IsChangePoint: Marked as a change because one of 4/6/2022 11:11:30 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -42.54586044524999 (T) = (0 -71815.0535166913) / Math.Sqrt((2025730.8379846802 / (37)) + (385044.8524190948 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.1813951139523011 = (60788.34478706913 - 71815.0535166913) / 60788.34478706913 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 101.86718141233767 > 95.54391778758875.
IsChangePoint: Marked as a change because one of 3/18/2022 7:38:50 AM, 4/6/2022 1:01:14 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -35.48605983171002 (T) = (0 -102484.75637328616) / Math.Sqrt((2365733.44482041 / (37)) + (823612.661316888 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.11628222787677776 = (91809.00117725342 - 102484.75637328616) / 91809.00117725342 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 88.41445896388217 > 79.70534152680308.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/11/2022 7:02:26 PM, 5/17/2022 3:30:02 AM falls between 5/8/2022 12:35:08 PM and 5/17/2022 3:30:02 AM.
IsRegressionStdDev: Marked as regression because -52.114040337106 (T) = (0 -87778.56423858648) / Math.Sqrt((1551252.8914258825 / (37)) + (292123.1711855977 / (31))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (31) - 2, .025) and -0.1554471231627319 = (75969.3476914078 - 87778.56423858648) / 75969.3476914078 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: kunalspathak
Labels:

area-CodeGen-coreclr, untriaged, refs/heads/main, ubuntu 18.04, RunKind=micro, Regression, CoreClr, x64
Milestone: -

@kunalspathak
Copy link
Member

I verified the asmdiffs and I don't see any difference introduced by 336a5d2...9d972ed. @DrewScoggins - Did anything change about the machines where the benchmarks run or the order in which they run? Is it possible to get ETLs for them?

@JulieLeeMSFT JulieLeeMSFT removed the untriaged New issue has not been triaged by the area owner label May 20, 2022
@JulieLeeMSFT JulieLeeMSFT added this to the 7.0.0 milestone May 20, 2022
@kunalspathak kunalspathak mentioned this issue Jun 2, 2022
Merged
@kunalspathak
Copy link
Member

Possibly this will be fixed by #70144

@kunalspathak
Copy link
Member

kunalspathak commented Jun 6, 2022 •
edited
Loading

As mentioned, all the benchmarks regained their performance.

image

image

image

This one is interesting. As part of #68869 when we accidently started to not align any loops, we again see the instability in the benchmarks while after fixing the bug in #70144, we see not only performance improves but stability comes back.

image

@dakersnar dakersnar mentioned this issue Jun 10, 2022
Closed
14 tasks
@ghost ghost locked as resolved and limited conversation to collaborators Jul 6, 2022
@jeffhandley jeffhandley added runtime-coreclr specific to the CoreCLR runtime and removed CoreClr labels Dec 28, 2022
@jeffhandley jeffhandley added os-linux Linux OS (any supported distro) arch-x64 and removed refs/heads/main labels Dec 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@kunalspathak kunalspathak

Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
7 participants
@EgorBo @jeffhandley @DrewScoggins @jeffschwMSFT @kunalspathak @dakersnar @JulieLeeMSFT