Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 8 Regressions on 10/26/2022 10:56:11 AM #9358

Closed
performanceautofiler bot opened this issue Nov 1, 2022 · 1 comment
Closed

[Perf] Linux/x64: 8 Regressions on 10/26/2022 10:56:11 AM #9358

performanceautofiler bot opened this issue Nov 1, 2022 · 1 comment
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-aot os-linux perf-regression runtime-mono untriaged

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Nov 1, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8ea8e67dc20689aa2e4c738a2f84ea3483ec872
Compare 129982ef61894433867f0ea593281fc069284ade
Diff Diff

Regressions in System.Collections.CreateAddAndClear<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableList - Duration of single invocation 188.11 μs 200.11 μs 1.06 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CreateAddAndClear&lt;String&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.CreateAddAndClear<String>.ImmutableList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 200.1137718482906 > 197.51161030611286.
IsChangePoint: Marked as a change because one of 9/2/2022 11:09:40 AM, 9/10/2022 11:12:52 PM, 9/27/2022 4:04:11 PM, 10/26/2022 7:11:30 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -32.60040103042265 (T) = (0 -199368.53266580938) / Math.Sqrt((1883608.5319391463 / (25)) + (905108.042070416 / (27))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (27) - 2, .025) and -0.057029739458901677 = (188612.03731871067 - 199368.53266580938) / 188612.03731871067 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline a8ea8e67dc20689aa2e4c738a2f84ea3483ec872
Compare 129982ef61894433867f0ea593281fc069284ade
Diff Diff

Regressions in System.MathBenchmarks.Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ILogB - Duration of single invocation 15.83 μs 18.93 μs 1.20 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.MathBenchmarks.Single*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.MathBenchmarks.Single.ILogB

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 18.930278205432714 > 16.630938718345423.
IsChangePoint: Marked as a change because one of 8/22/2022 5:37:03 PM, 9/2/2022 11:09:40 AM, 9/10/2022 11:12:52 PM, 9/21/2022 9:54:19 PM, 10/11/2022 8:42:28 PM, 10/18/2022 9:26:37 AM, 10/26/2022 7:11:30 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -55.06549681098241 (T) = (0 -18876.31699056505) / Math.Sqrt((66394.09030441183 / (25)) + (4516.691015791454 / (27))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (27) - 2, .025) and -0.18342683875414798 = (15950.556783414751 - 18876.31699056505) / 15950.556783414751 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8ea8e67dc20689aa2e4c738a2f84ea3483ec872
Compare 129982ef61894433867f0ea593281fc069284ade
Diff Diff

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ElementAt - Duration of single invocation 185.18 ns 194.81 ns 1.05 0.11 False
SingleWithPredicate_FirstElementMatches - Duration of single invocation 611.80 ns 700.14 ns 1.14 0.30 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Linq.Tests.Perf_Enumerable.ElementAt(input: IEnumerable)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 194.80644470453856 > 194.40733443314818.
IsChangePoint: Marked as a change because one of 9/10/2022 11:12:52 PM, 9/26/2022 2:03:14 AM, 10/26/2022 7:11:30 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -20.06434627146081 (T) = (0 -195.82237675807133) / Math.Sqrt((3.5704916021855806 / (25)) + (2.0421753721969305 / (27))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (27) - 2, .025) and -0.05029876606458814 = (186.44445093638168 - 195.82237675807133) / 186.44445093638168 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_FirstElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 700.1433841682377 > 643.5025964570984.
IsChangePoint: Marked as a change because one of 10/21/2022 5:53:28 PM, 10/25/2022 2:34:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -4.425219910196433 (T) = (0 -726.8705354920702) / Math.Sqrt((4691.966427674501 / (21)) + (481.7150273355697 / (31))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (31) - 2, .025) and -0.1038894863754684 = (658.46313826096 - 726.8705354920702) / 658.46313826096 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8ea8e67dc20689aa2e4c738a2f84ea3483ec872
Compare e6bac1e15f3ea795c84cbd009eb8b77da460c9e1
Diff Diff

Regressions in System.Collections.IterateFor<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableSortedSet - Duration of single invocation 23.09 μs 25.24 μs 1.09 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IterateFor&lt;String&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.IterateFor<String>.ImmutableSortedSet(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 25.243890988996245 > 24.511334556789677.
IsChangePoint: Marked as a change because one of 9/10/2022 11:12:52 PM, 9/29/2022 8:01:59 PM, 10/26/2022 7:11:30 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -41.99357451902429 (T) = (0 -25455.00557885572) / Math.Sqrt((19321.133477016367 / (24)) + (48722.86059137889 / (26))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (26) - 2, .025) and -0.09335967522988473 = (23281.456372994246 - 25455.00557885572) / 23281.456372994246 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8ea8e67dc20689aa2e4c738a2f84ea3483ec872
Compare 129982ef61894433867f0ea593281fc069284ade
Diff Diff

Regressions in System.Collections.ContainsTrueComparer<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedSet - Duration of single invocation 48.67 μs 52.02 μs 1.07 0.06 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsTrueComparer&lt;Int32&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.ContainsTrueComparer<Int32>.SortedSet(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 52.02470519774012 > 50.710285081650085.
IsChangePoint: Marked as a change because one of 8/30/2022 12:00:50 PM, 9/2/2022 11:09:40 AM, 10/15/2022 5:22:23 AM, 10/26/2022 7:11:30 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -41.26075930788213 (T) = (0 -51754.98944666404) / Math.Sqrt((79668.16651748648 / (25)) + (108228.6326066491 / (27))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (27) - 2, .025) and -0.07252964767747375 = (48255.06647647242 - 51754.98944666404) / 48255.06647647242 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8ea8e67dc20689aa2e4c738a2f84ea3483ec872
Compare 129982ef61894433867f0ea593281fc069284ade
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexerCheckPathLength - Duration of single invocation 151.99 ns 169.48 ns 1.12 0.04 False
Split - Duration of single invocation 73.07 ns 82.16 ns 1.12 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Tests.Perf_String.IndexerCheckPathLength

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 169.47727555587142 > 159.56902388972648.
IsChangePoint: Marked as a change because one of 8/30/2022 6:17:03 PM, 9/2/2022 11:09:40 AM, 9/24/2022 1:05:07 PM, 10/26/2022 7:11:30 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -10.5218729625722 (T) = (0 -169.47194786658727) / Math.Sqrt((101.56977927287534 / (25)) + (0.04939320572452778 / (27))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (27) - 2, .025) and -0.14308110406080463 = (148.25890067164687 - 169.47194786658727) / 148.25890067164687 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 82.15529043362841 > 77.62707227102747.
IsChangePoint: Marked as a change because one of 8/31/2022 10:21:55 AM, 9/6/2022 5:23:23 PM, 10/5/2022 3:28:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -20.74206247182214 (T) = (0 -82.69757771395193) / Math.Sqrt((1.1682459328745716 / (22)) + (2.647640828093604 / (30))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (30) - 2, .025) and -0.10411989187432266 = (74.89909232009838 - 82.69757771395193) / 74.89909232009838 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@naricc
Copy link

naricc commented Nov 2, 2022

It seems unlikely any of the changes in the range could have causes this.

@naricc naricc closed this as completed Nov 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-aot os-linux perf-regression runtime-mono untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@naricc