Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions in Math.Min/Max benchmarks #88096

Closed
performanceautofiler bot opened this issue Jun 27, 2023 · 11 comments · Fixed by #88106
Closed

Regressions in Math.Min/Max benchmarks #88096

performanceautofiler bot opened this issue Jun 27, 2023 · 11 comments · Fixed by #88106
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
8.0.0

Comments

@performanceautofiler
Copy link

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4542e09ba0494eb29c86c6255fcba7fd9db897b7
Compare 3a303b8e0a67e3e330c26b4a305caa407f66e443
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in BilinearTest

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Interpol_Scalar - Duration of single invocation 6.97 μs 11.39 μs 1.64 0.01 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'BilinearTest*'

Payloads

Baseline
Compare

Histogram

BilinearTest.Interpol_Scalar

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.392019824289877 > 7.3080500558187635.
IsChangePoint: Marked as a change because one of 6/19/2023 11:51:49 PM, 6/26/2023 6:52:13 PM falls between 6/17/2023 6:37:57 PM and 6/26/2023 6:52:13 PM.
IsRegressionStdDev: Marked as regression because -212.27980464814564 (T) = (0 -11425.0926797762) / Math.Sqrt((3264.2035944192285 / (19)) + (6315.325408949749 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -0.646241001817783 = (6940.1094172485 - 11425.0926797762) / 6940.1094172485 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4542e09ba0494eb29c86c6255fcba7fd9db897b7
Compare 3a303b8e0a67e3e330c26b4a305caa407f66e443
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.MathBenchmarks.Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AcosPi - Duration of single invocation 33.16 μs 34.85 μs 1.05 0.05 False
AsinPi - Duration of single invocation 26.10 μs 27.80 μs 1.07 0.06 False
Min - Duration of single invocation 4.36 μs 9.79 μs 2.25 0.01 True
Max - Duration of single invocation 4.36 μs 9.82 μs 2.25 0.05 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Single*'

Payloads

Baseline
Compare

Histogram

System.MathBenchmarks.Single.AcosPi

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.85185961395694 > 34.81215898305084.
IsChangePoint: Marked as a change because one of 4/12/2023 9:47:42 AM, 6/19/2023 11:51:49 PM, 6/26/2023 6:52:13 PM falls between 6/17/2023 6:37:57 PM and 6/26/2023 6:52:13 PM.
IsRegressionStdDev: Marked as regression because -275.73947503992264 (T) = (0 -34843.3414755857) / Math.Sqrt((146.78381967633746 / (19)) + (696.340970241494 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -0.051285952678343606 = (33143.543283172294 - 34843.3414755857) / 33143.543283172294 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.MathBenchmarks.Single.AsinPi

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.803446097385716 > 27.411496575261655.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 6/19/2023 11:51:49 PM, 6/26/2023 6:52:13 PM falls between 6/17/2023 6:37:57 PM and 6/26/2023 6:52:13 PM.
IsRegressionStdDev: Marked as regression because -11.07046276908435 (T) = (0 -28015.593903971618) / Math.Sqrt((2300.110939347 / (19)) + (665512.5293408092 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -0.07222234628146819 = (26128.530151541225 - 28015.593903971618) / 26128.530151541225 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.MathBenchmarks.Single.Min

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.793916284099673 > 4.574769058553263.
IsChangePoint: Marked as a change because one of 6/19/2023 11:51:49 PM, 6/26/2023 6:52:13 PM falls between 6/17/2023 6:37:57 PM and 6/26/2023 6:52:13 PM.
IsRegressionStdDev: Marked as regression because -4077.942787750875 (T) = (0 -9793.360710345663) / Math.Sqrt((8.890539597199659 / (19)) + (30.107403114004754 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -1.247532194273326 = (4357.383949960308 - 9793.360710345663) / 4357.383949960308 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.MathBenchmarks.Single.Max

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.816313532156284 > 4.575135782515241.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/19/2023 11:51:49 PM, 6/26/2023 6:52:13 PM falls between 6/17/2023 6:37:57 PM and 6/26/2023 6:52:13 PM.
IsRegressionStdDev: Marked as regression because -2560.568103936479 (T) = (0 -9796.268681764557) / Math.Sqrt((4.905999652468138 / (19)) + (97.86250809145457 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -1.2486135858622394 = (4356.5816480683325 - 9796.268681764557) / 4356.5816480683325 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4542e09ba0494eb29c86c6255fcba7fd9db897b7
Compare 3a303b8e0a67e3e330c26b4a305caa407f66e443
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.MathBenchmarks.Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Min - Duration of single invocation 4.36 μs 9.80 μs 2.25 0.02 True
Max - Duration of single invocation 4.37 μs 9.80 μs 2.24 0.06 True

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Double*'

Payloads

Baseline
Compare

Histogram

System.MathBenchmarks.Double.Min

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.797353483970342 > 4.580646210246807.
IsChangePoint: Marked as a change because one of 6/19/2023 11:51:49 PM, 6/26/2023 6:52:13 PM falls between 6/17/2023 6:37:57 PM and 6/26/2023 6:52:13 PM.
IsRegressionStdDev: Marked as regression because -4108.113806202843 (T) = (0 -9799.234437855583) / Math.Sqrt((6.967540554707664 / (19)) + (31.835250218845825 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -1.2457873252163145 = (4363.384870787673 - 9799.234437855583) / 4363.384870787673 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.MathBenchmarks.Double.Max

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.80185848354232 > 4.582261803762309.
IsChangePoint: Marked as a change because one of 4/25/2023 1:57:48 PM, 5/19/2023 10:30:38 AM, 6/19/2023 11:51:49 PM, 6/26/2023 6:52:13 PM falls between 6/17/2023 6:37:57 PM and 6/26/2023 6:52:13 PM.
IsRegressionStdDev: Marked as regression because -2954.2342013487714 (T) = (0 -9798.381123671425) / Math.Sqrt((30.325791946983806 / (18)) + (39.05060689527017 / (23))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (23) - 2, .025) and -1.244769051994068 = (4364.984057031324 - 9798.381123671425) / 4364.984057031324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-x64 os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels Jun 27, 2023
@EgorBo EgorBo changed the title [Perf] Linux/x64: 7 Regressions on 6/20/2023 3:32:08 AM Regressions in Math.Min/Max benchmarks Jun 27, 2023
@EgorBo EgorBo transferred this issue from dotnet/perf-autofiling-issues Jun 27, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jun 27, 2023
@EgorBo EgorBo added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Jun 27, 2023
@ghost
Copy link

ghost commented Jun 27, 2023

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4542e09ba0494eb29c86c6255fcba7fd9db897b7
Compare 3a303b8e0a67e3e330c26b4a305caa407f66e443
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in BilinearTest

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Interpol_Scalar - Duration of single invocation 6.97 μs 11.39 μs 1.64 0.01 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'BilinearTest*'

Payloads

Baseline
Compare

Histogram

BilinearTest.Interpol_Scalar

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.392019824289877 > 7.3080500558187635.
IsChangePoint: Marked as a change because one of 6/19/2023 11:51:49 PM, 6/26/2023 6:52:13 PM falls between 6/17/2023 6:37:57 PM and 6/26/2023 6:52:13 PM.
IsRegressionStdDev: Marked as regression because -212.27980464814564 (T) = (0 -11425.0926797762) / Math.Sqrt((3264.2035944192285 / (19)) + (6315.325408949749 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -0.646241001817783 = (6940.1094172485 - 11425.0926797762) / 6940.1094172485 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4542e09ba0494eb29c86c6255fcba7fd9db897b7
Compare 3a303b8e0a67e3e330c26b4a305caa407f66e443
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.MathBenchmarks.Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AcosPi - Duration of single invocation 33.16 μs 34.85 μs 1.05 0.05 False
AsinPi - Duration of single invocation 26.10 μs 27.80 μs 1.07 0.06 False
Min - Duration of single invocation 4.36 μs 9.79 μs 2.25 0.01 True
Max - Duration of single invocation 4.36 μs 9.82 μs 2.25 0.05 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Single*'

Payloads

Baseline
Compare

Histogram

System.MathBenchmarks.Single.AcosPi

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.85185961395694 > 34.81215898305084.
IsChangePoint: Marked as a change because one of 4/12/2023 9:47:42 AM, 6/19/2023 11:51:49 PM, 6/26/2023 6:52:13 PM falls between 6/17/2023 6:37:57 PM and 6/26/2023 6:52:13 PM.
IsRegressionStdDev: Marked as regression because -275.73947503992264 (T) = (0 -34843.3414755857) / Math.Sqrt((146.78381967633746 / (19)) + (696.340970241494 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -0.051285952678343606 = (33143.543283172294 - 34843.3414755857) / 33143.543283172294 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.MathBenchmarks.Single.AsinPi

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.803446097385716 > 27.411496575261655.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 6/19/2023 11:51:49 PM, 6/26/2023 6:52:13 PM falls between 6/17/2023 6:37:57 PM and 6/26/2023 6:52:13 PM.
IsRegressionStdDev: Marked as regression because -11.07046276908435 (T) = (0 -28015.593903971618) / Math.Sqrt((2300.110939347 / (19)) + (665512.5293408092 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -0.07222234628146819 = (26128.530151541225 - 28015.593903971618) / 26128.530151541225 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.MathBenchmarks.Single.Min

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.793916284099673 > 4.574769058553263.
IsChangePoint: Marked as a change because one of 6/19/2023 11:51:49 PM, 6/26/2023 6:52:13 PM falls between 6/17/2023 6:37:57 PM and 6/26/2023 6:52:13 PM.
IsRegressionStdDev: Marked as regression because -4077.942787750875 (T) = (0 -9793.360710345663) / Math.Sqrt((8.890539597199659 / (19)) + (30.107403114004754 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -1.247532194273326 = (4357.383949960308 - 9793.360710345663) / 4357.383949960308 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.MathBenchmarks.Single.Max

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.816313532156284 > 4.575135782515241.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/19/2023 11:51:49 PM, 6/26/2023 6:52:13 PM falls between 6/17/2023 6:37:57 PM and 6/26/2023 6:52:13 PM.
IsRegressionStdDev: Marked as regression because -2560.568103936479 (T) = (0 -9796.268681764557) / Math.Sqrt((4.905999652468138 / (19)) + (97.86250809145457 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -1.2486135858622394 = (4356.5816480683325 - 9796.268681764557) / 4356.5816480683325 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 4542e09ba0494eb29c86c6255fcba7fd9db897b7
Compare 3a303b8e0a67e3e330c26b4a305caa407f66e443
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.MathBenchmarks.Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Min - Duration of single invocation 4.36 μs 9.80 μs 2.25 0.02 True
Max - Duration of single invocation 4.37 μs 9.80 μs 2.24 0.06 True

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Double*'

Payloads

Baseline
Compare

Histogram

System.MathBenchmarks.Double.Min

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.797353483970342 > 4.580646210246807.
IsChangePoint: Marked as a change because one of 6/19/2023 11:51:49 PM, 6/26/2023 6:52:13 PM falls between 6/17/2023 6:37:57 PM and 6/26/2023 6:52:13 PM.
IsRegressionStdDev: Marked as regression because -4108.113806202843 (T) = (0 -9799.234437855583) / Math.Sqrt((6.967540554707664 / (19)) + (31.835250218845825 / (23))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (23) - 2, .025) and -1.2457873252163145 = (4363.384870787673 - 9799.234437855583) / 4363.384870787673 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.MathBenchmarks.Double.Max

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.80185848354232 > 4.582261803762309.
IsChangePoint: Marked as a change because one of 4/25/2023 1:57:48 PM, 5/19/2023 10:30:38 AM, 6/19/2023 11:51:49 PM, 6/26/2023 6:52:13 PM falls between 6/17/2023 6:37:57 PM and 6/26/2023 6:52:13 PM.
IsRegressionStdDev: Marked as regression because -2954.2342013487714 (T) = (0 -9798.381123671425) / Math.Sqrt((30.325791946983806 / (18)) + (39.05060689527017 / (23))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (23) - 2, .025) and -1.244769051994068 = (4364.984057031324 - 9798.381123671425) / 4364.984057031324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

os-linux, tenet-performance, tenet-performance-benchmarks, arch-x64, area-CodeGen-coreclr, untriaged, runtime-coreclr
Milestone: -

@EgorBo
Copy link
Member

EgorBo commented Jun 27, 2023

Regressed via #87641 cc @tannergooding

@EgorBo EgorBo added this to the 8.0.0 milestone Jun 27, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Jun 27, 2023
@EgorBo
Copy link
Member

EgorBo commented Jun 27, 2023 •
edited
Loading

Same on win-x64:

@tannergooding
Copy link
Member

@EgorBo, what hardware are the TigerUbuntu machines?

The only change for Min/Max on pre-AVX512 hardware is that rather than always returning nullptr (and therefore always keeping it as a GT_CALL) when neither input is constant, it now matches other intrinsics and returns a GT_INTRINSIC node which allows CSE. It will then be converted back to a call in rationalization.

@EgorBo
Copy link
Member

EgorBo commented Jun 27, 2023

@EgorBo, what hardware are the TigerUbuntu machines?

I am not sure but Tiger should stand for Intel (without avx-512)

@EgorBo
Copy link
Member

EgorBo commented Jun 27, 2023

Reproduces on OwlWindows (AMD, no avx-512) too

@tannergooding
Copy link
Member

Sounds like a general CSE issue then and likely one that impacts other GT_INTRINSIC nodes.

Will take a look when I'm back next week.

@EgorBo
Copy link
Member

EgorBo commented Jun 27, 2023

@tannergooding it seems that Math.Min/Max is expected to be inlined, but if you convert it to GT_INTRINSIC and then rewrite to user-call (if you don't manage to expand it) you won't be able to inline it.

@EgorBo
Copy link
Member

EgorBo commented Jun 27, 2023

Codegen on non-avx512 hw:

; Method Prog+Double:MinTest()
G_M35241_IG01:  ;; offset=0000H
       push     rsi
       sub      rsp, 96
       vzeroupper 
       vmovaps  xmmword ptr [rsp+50H], xmm6
       vmovaps  xmmword ptr [rsp+40H], xmm7
       vmovaps  xmmword ptr [rsp+30H], xmm8
       vmovaps  xmmword ptr [rsp+20H], xmm9
						;; size=32 bbWeight=1 PerfScore 10.25

G_M35241_IG02:  ;; offset=0020H
       vxorps   xmm6, xmm6, xmm6
       vmovsd   xmm7, qword ptr [reloc @RWD00]
       vmovsd   xmm8, qword ptr [reloc @RWD08]
       xor      esi, esi
       vmovsd   xmm9, qword ptr [reloc @RWD16]
       align    [2 bytes for IG03]
						;; size=32 bbWeight=1 PerfScore 9.83

G_M35241_IG03:  ;; offset=0040H
       vsubsd   xmm8, xmm8, xmm9
       vmovsd   xmm0, qword ptr [reloc @RWD00]
       vmovaps  xmm1, xmm8
       call     [System.Math:Min(double,double):double]
       vaddsd   xmm6, xmm0, xmm6
       inc      esi
       cmp      esi, 0x1388
       jl       SHORT G_M35241_IG03
						;; size=38 bbWeight=4 PerfScore 55.00

G_M35241_IG04:  ;; offset=0066H
       vsubsd   xmm0, xmm7, xmm6
       vandps   xmm0, xmm0, xmmword ptr [reloc @RWD32]
       vucomisd xmm0, qword ptr [reloc @RWD48]
       ja       SHORT G_M35241_IG06
						;; size=22 bbWeight=1 PerfScore 10.00

G_M35241_IG05:  ;; offset=007CH
       vmovaps  xmm6, xmmword ptr [rsp+50H]
       vmovaps  xmm7, xmmword ptr [rsp+40H]
       vmovaps  xmm8, xmmword ptr [rsp+30H]
       vmovaps  xmm9, xmmword ptr [rsp+20H]
       add      rsp, 96
       pop      rsi
       ret      
						;; size=30 bbWeight=1 PerfScore 17.75

G_M35241_IG06:  ;; offset=009AH
       mov      rcx, 0x7FF9291DB1C8      ; System.Exception
       call     CORINFO_HELP_NEWSFAST
       mov      rsi, rax
       mov      rcx, rsi
       call     [System.Exception:.ctor():this]
       mov      rcx, rsi
       call     CORINFO_HELP_THROW
       int3     
						;; size=36 bbWeight=0 PerfScore 0.00
RWD00  	dq	3FF0000000000000h	;            1
RWD08  	dq	3FF001A36E2EB1C4h	;       1.0004
RWD16  	dq	3F3A36E2EB1C432Dh	;       0.0004
RWD24  	dd	00000000h, 00000000h
RWD32  	dq	7FFFFFFFFFFFFFFFh, 7FFFFFFFFFFFFFFFh
RWD48  	dq	3E7FF19E23A836FCh	;     1.19e-07
; Total bytes of code: 190

@tannergooding
Copy link
Member

but if you convert it to GT_INTRINSIC and then rewrite to user-call (if you don't manage to expand it) you won't be able to inline it.

That's an unfortunate side effect for a number of math APIs which are implemented in managed but which we want to CSE...

Given we track the original call handle, how hard would it be to have the inliner look at GT_INTRINSIC nodes as well or instead to allow CSE to work on some subset of GT_CALL nodes?

@SingleAccretion
Copy link
Contributor

It would not be "hard" to allow VNing/CSEing of user calls, but it would be somewhat involved because code assumes in a number of places that only helper calls can be "understood".

Inlining GT_INTRINSIC is likely a non-starter.

@tannergooding tannergooding mentioned this issue Jun 27, 2023
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Jun 27, 2023
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Jun 27, 2023
@AndyAyersMS AndyAyersMS mentioned this issue Jul 26, 2023
Closed
@ghost ghost locked as resolved and limited conversation to collaborators Jul 28, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
8.0.0
Development

Successfully merging a pull request may close this issue.

Ensure that we can still inline Math.Max and Math.Min on downlevel hardware tannergooding/runtime
3 participants
@EgorBo @tannergooding @SingleAccretion