[Perf] Linux/x64: 10 Regressions on 11/2/2022 9:35:40 AM

[performanceautofiler]

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	d3a3eda6f4660e85f7a603b624b43993529be377
Compare	13d4e6de2c4ecd1414bd29d5240fb6fe2e33b9fa
Diff	Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Int64>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ZeroBenchmark - Duration of single invocation	16.83 ns	22.45 ns	1.33	0.60	False
XorBenchmark - Duration of single invocation	34.08 ns	80.04 ns	2.35	0.53	False
BitwiseOrBenchmark - Duration of single invocation	35.46 ns	74.57 ns	2.10	0.59	False
BitwiseOrOperatorBenchmark - Duration of single invocation	20.26 ns	75.36 ns	3.72	0.49	False
AndNotBenchmark - Duration of single invocation	104.87 ns	123.62 ns	1.18	0.48	False
ExclusiveOrOperatorBenchmark - Duration of single invocation	20.22 ns	75.38 ns	3.73	0.60	False
BitwiseAndBenchmark - Duration of single invocation	30.51 ns	75.15 ns	2.46	0.58	False
ConditionalSelectBenchmark - Duration of single invocation	136.46 ns	255.36 ns	1.87	0.32	False
CountBenchmark - Duration of single invocation	11.21 ns	22.36 ns	2.00	0.62	False
BitwiseAndOperatorBenchmark - Duration of single invocation	19.36 ns	76.54 ns	3.95	0.57	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf<Int64>*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Int64>.ZeroBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 22.4483722721432 > 17.476264852084494.
IsChangePoint: Marked as a change because one of 11/2/2022 4:14:20 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -8.899564638649577 (T) = (0 -23.254950236775784) / Math.Sqrt((5.264488054241241 / (21)) + (2.0272157440674956 / (14))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (14) - 2, .025) and -0.31695019005106234 = (17.65818511015524 - 23.254950236775784) / 17.65818511015524 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf<Int64>.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 80.04102511127428 > 32.68352557980007.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -64.20972014527702 (T) = (0 -76.94984932106955) / Math.Sqrt((4.7832279291132185 / (23)) + (3.6270067204260923 / (13))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (13) - 2, .025) and -1.3940316485912443 = (32.14236928168861 - 76.94984932106955) / 32.14236928168861 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf<Int64>.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 74.56592184108571 > 35.54342510918686.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -67.14114466606144 (T) = (0 -76.00487942031609) / Math.Sqrt((4.201437541761191 / (23)) + (3.0092699851666556 / (14))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (14) - 2, .025) and -1.2575040895157024 = (33.6676596836736 - 76.00487942031609) / 33.6676596836736 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf<Int64>.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 75.36184781669719 > 23.05393393401559.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -121.40202822258749 (T) = (0 -75.79304445061578) / Math.Sqrt((2.4622349121574882 / (21)) + (1.1483160501271223 / (14))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (14) - 2, .025) and -2.5090462347803695 = (21.599329099566468 - 75.79304445061578) / 21.599329099566468 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf<Int64>.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 123.62145458438194 > 121.26691189076104.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -8.779713996301625 (T) = (0 -124.93149417398605) / Math.Sqrt((31.50663376014741 / (23)) + (5.285457178038058 / (13))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (13) - 2, .025) and -0.10334605339684377 = (113.22965608963987 - 124.93149417398605) / 113.22965608963987 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf<Int64>.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 75.38202304566649 > 21.91898292907328.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -159.61420343477627 (T) = (0 -75.51422192639849) / Math.Sqrt((2.2779828510863975 / (22)) + (0.15297769045891185 / (14))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (14) - 2, .025) and -2.5105103816897905 = (21.510895486954688 - 75.51422192639849) / 21.510895486954688 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf<Int64>.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 75.15317290351233 > 33.286807875829005.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -57.023904301479305 (T) = (0 -76.81931709197855) / Math.Sqrt((3.129470243914856 / (22)) + (6.667887350925081 / (14))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (14) - 2, .025) and -1.4027027105724101 = (31.97204412928698 - 76.81931709197855) / 31.97204412928698 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf<Int64>.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 255.35844204963604 > 143.02776567876907.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -81.8712647127991 (T) = (0 -258.8000598513955) / Math.Sqrt((39.851611620128594 / (23)) + (5.459951612639187 / (14))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (14) - 2, .025) and -0.8549514683126175 = (139.5185072344866 - 258.8000598513955) / 139.5185072344866 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf<Int64>.CountBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 22.35928558414351 > 11.901869289723347.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -22.85051236161695 (T) = (0 -21.405555049427953) / Math.Sqrt((0.8409748114078303 / (22)) + (1.9154594965714205 / (13))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (13) - 2, .025) and -0.8513599686774705 = (11.562070808260552 - 21.405555049427953) / 11.562070808260552 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf<Int64>.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 76.5437649604222 > 21.085312306268115.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -156.8085760482449 (T) = (0 -75.87966553444528) / Math.Sqrt((1.8955285596861697 / (23)) + (0.5158514649016369 / (13))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (13) - 2, .025) and -2.598336931438522 = (21.087426491801743 - 75.87966553444528) / 21.087426491801743 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[radekdoulik]

Same as #9637 likely caused by dotnet/runtime#77562

[tannergooding]

As per dotnet/runtime#78163 (comment), the fixes taken here are required for correctness.

These tests should effectively be ignored on Mono until the SIMD work is completed and IsHardwareAccelerated starts reporting true. The software fallbacks are not expected to be used in production scenarios and exist purely for validation purposes and to support scenarios like the debugger.