Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Windows/x64: 15 Regressions in Microsoft.Extensions.DependencyInjection #89104

Closed
performanceautofiler bot opened this issue Jul 18, 2023 · 10 comments
Closed

[Perf] Windows/x64: 15 Regressions in Microsoft.Extensions.DependencyInjection #89104

performanceautofiler bot opened this issue Jul 18, 2023 · 10 comments
Assignees
@benjaminpetit
Labels
arch-x64 area-Extensions-DependencyInjection os-windows runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
8.0.0

Comments

@performanceautofiler
Copy link

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline b9c99648783d0c28afdee394a93516c0bd228165
Compare 9409e2d5019dc46a7a8fd1c59b19a95ddaa3106e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.DependencyInjection.TimeToFirstService

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
Transient - Duration of single invocation
📝 - Benchmark Source		 2.07 μs 2.81 μs 1.36 0.03 False
Scoped - Duration of single invocation
📝 - Benchmark Source		 2.21 μs 3.74 μs 1.70 0.01 False
Singleton - Duration of single invocation
📝 - Benchmark Source		 2.14 μs 2.86 μs 1.33 0.03 False
Transient - Duration of single invocation
📝 - Benchmark Source		 2.07 μs 2.79 μs 1.35 0.02 False
Singleton - Duration of single invocation
📝 - Benchmark Source		 2.14 μs 2.86 μs 1.34 0.01 False
Singleton - Duration of single invocation
📝 - Benchmark Source		 2.15 μs 2.86 μs 1.33 0.02 False
Transient - Duration of single invocation
📝 - Benchmark Source		 2.06 μs 2.81 μs 1.36 0.01 False
Transient - Duration of single invocation
📝 - Benchmark Source		 2.04 μs 3.04 μs 1.49 0.02 False
Scoped - Duration of single invocation
📝 - Benchmark Source		 2.19 μs 3.68 μs 1.68 0.01 False
Singleton - Duration of single invocation
📝 - Benchmark Source		 2.12 μs 2.85 μs 1.34 0.01 False
Scoped - Duration of single invocation
📝 - Benchmark Source		 2.18 μs 4.43 μs 2.03 0.01 False
Scoped - Duration of single invocation
📝 - Benchmark Source		 2.22 μs 3.70 μs 1.67 0.02 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.TimeToFirstService*'

Payloads

Baseline
Compare

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Transient(Mode: "ILEmit")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.8074656687937947 > 2.169533256426224.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -58.719093257450126 (T) = (0 -2792.851037422641) / Math.Sqrt((352.60395059866795 / (25)) + (543.8548409516927 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.3469069695147478 = (2073.529279033151 - 2792.851037422641) / 2073.529279033151 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Scoped(Mode: "Expressions")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.7449060919169086 > 2.313329304203398.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -54.923040510076426 (T) = (0 -3697.0819345898108) / Math.Sqrt((537.825920760676 / (25)) + (2789.87435868882 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.662069476493108 = (2224.384712479341 - 3697.0819345898108) / 2224.384712479341 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "Dynamic")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.85680345824012 > 2.252353329493443.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -31.263725112178555 (T) = (0 -2830.783672344858) / Math.Sqrt((432.5965154995103 / (25)) + (1861.7552338641817 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.3204047668454227 = (2143.875683748007 - 2830.783672344858) / 2143.875683748007 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Transient(Mode: "Dynamic")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.7852518123062144 > 2.1674703674283275.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -39.18403923643714 (T) = (0 -2796.1699528407926) / Math.Sqrt((601.8852429331239 / (25)) + (1258.76112347285 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.347574743169357 = (2074.964648168226 - 2796.1699528407926) / 2074.964648168226 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "Runtime")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.8638615516169157 > 2.236946583419647.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -13.674900818646606 (T) = (0 -2867.3547131966916) / Math.Sqrt((693.2125990057299 / (25)) + (10946.757199621194 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.33467176812110977 = (2148.3594556234775 - 2867.3547131966916) / 2148.3594556234775 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "ILEmit")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.860767285628019 > 2.2581899871910105.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -28.459806100580693 (T) = (0 -2814.6439064723313) / Math.Sqrt((394.51806269517533 / (25)) + (1602.6110112794086 / (3))) is less than -2.055529438636913 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (3) - 2, .025) and -0.3108366661755923 = (2147.211761083968 - 2814.6439064723313) / 2147.211761083968 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Transient(Mode: "Expressions")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.8059250772847997 > 2.156217797528226.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -109.30110293957469 (T) = (0 -2792.969034167149) / Math.Sqrt((582.6300813353151 / (25)) + (82.0607472504304 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.3496327992981993 = (2069.428836954374 - 2792.969034167149) / 2069.428836954374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Transient(Mode: "Runtime")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.039595297918162 > 2.1777183476809574.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -11.301501377442008 (T) = (0 -2841.6026985209705) / Math.Sqrt((474.54920924768743 / (25)) + (18310.227520502194 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.3691923776596196 = (2075.3860048345896 - 2841.6026985209705) / 2075.3860048345896 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Scoped(Mode: "ILEmit")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.676834172199614 > 2.3172891684729664.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -35.680484435747516 (T) = (0 -3670.6950226613326) / Math.Sqrt((1349.1500413067165 / (25)) + (6340.841753427479 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.6489291581903674 = (2226.108383388508 - 3670.6950226613326) / 2226.108383388508 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "Expressions")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.846683093547583 > 2.228866049088209.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -30.937862344587977 (T) = (0 -2832.288570055267) / Math.Sqrt((412.0683729006819 / (25)) + (1919.365963300643 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.321619113764339 = (2143.0444978872324 - 2832.288570055267) / 2143.0444978872324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Scoped(Mode: "Dynamic")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.426027441140356 > 2.3147573900974505.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -9.688500845345093 (T) = (0 -3906.8244227089135) / Math.Sqrt((390.3164674535471 / (25)) + (122453.70127502766 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.7668145123735033 = (2211.2250014635456 - 3906.8244227089135) / 2211.2250014635456 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Scoped(Mode: "Runtime")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.70442340470864 > 2.3019733705636787.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -11.795904307933982 (T) = (0 -3572.4373834456756) / Math.Sqrt((510.8867420414421 / (25)) + (53140.550141296815 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.6151848665431371 = (2211.7823522526583 - 3572.4373834456756) / 2211.7823522526583 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline b9c99648783d0c28afdee394a93516c0bd228165
Compare 9409e2d5019dc46a7a8fd1c59b19a95ddaa3106e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
CreateInstance - Duration of single invocation
📝 - Benchmark Source		 656.05 ns 1.13 μs 1.71 0.20 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark*'

Payloads

Baseline
Compare

Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark.CreateInstance

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1250784875784876 > 713.5991620104128.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -21.584278836659305 (T) = (0 -1139.3568733174607) / Math.Sqrt((1034.8133213404726 / (25)) + (1509.4652706212294 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.633100549870702 = (697.6648641797759 - 1139.3568733174607) / 697.6648641797759 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline b9c99648783d0c28afdee394a93516c0bd228165
Compare 9409e2d5019dc46a7a8fd1c59b19a95ddaa3106e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.DependencyInjection.GetServiceIEnumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
Scoped - Duration of single invocation
📝 - Benchmark Source		 31.63 ns 115.12 ns 3.64 0.27 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.GetServiceIEnumerable*'

Payloads

Baseline
Compare

Microsoft.Extensions.DependencyInjection.GetServiceIEnumerable.Scoped

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 115.11620215701704 > 33.50270896524814.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -35.204332932944325 (T) = (0 -118.81883672568196) / Math.Sqrt((4.423021630949348 / (25)) + (23.338142593740454 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -2.655525944638927 = (32.50389643655456 - 118.81883672568196) / 32.50389643655456 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline b9c99648783d0c28afdee394a93516c0bd228165
Compare 9409e2d5019dc46a7a8fd1c59b19a95ddaa3106e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.DependencyInjection.GetService

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
Scoped - Duration of single invocation
📝 - Benchmark Source		 31.57 ns 112.68 ns 3.57 0.19 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.GetService*'

Payloads

Baseline
Compare

Microsoft.Extensions.DependencyInjection.GetService.Scoped

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 112.679349578373 > 33.08173716511241.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -159.52245262501265 (T) = (0 -112.13476230036886) / Math.Sqrt((0.6216975524946721 / (25)) + (0.9190557309194967 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -2.54430120386857 = (31.638045372096162 - 112.13476230036886) / 31.638045372096162 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-x64 os-windows runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels Jul 18, 2023
@EgorBo EgorBo changed the title [Perf] Windows/x64: 15 Regressions on 7/13/2023 8:36:41 PM [Perf] Windows/x64: 15 Regressions in Microsoft.Extensions.DependencyInjection Jul 18, 2023
@EgorBo EgorBo removed the untriaged New issue has not been triaged by the area owner label Jul 18, 2023
@EgorBo EgorBo transferred this issue from dotnet/perf-autofiling-issues Jul 18, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jul 18, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jul 18, 2023
@EgorBo
Copy link
Member

EgorBo commented Jul 18, 2023

Regressed in #87183 cc @benjaminpetit

@EgorBo EgorBo added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Jul 18, 2023
@EgorBo
Copy link
Member

EgorBo commented Jul 18, 2023 •
edited
Loading

Same on Linux-x64:

Same on Windows-x64 (different machine):

@ghost
Copy link

ghost commented Jul 19, 2023

Tagging subscribers to this area: @dotnet/area-extensions-dependencyinjection
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline b9c99648783d0c28afdee394a93516c0bd228165
Compare 9409e2d5019dc46a7a8fd1c59b19a95ddaa3106e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.DependencyInjection.TimeToFirstService

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
Transient - Duration of single invocation
📝 - Benchmark Source		 2.07 μs 2.81 μs 1.36 0.03 False
Scoped - Duration of single invocation
📝 - Benchmark Source		 2.21 μs 3.74 μs 1.70 0.01 False
Singleton - Duration of single invocation
📝 - Benchmark Source		 2.14 μs 2.86 μs 1.33 0.03 False
Transient - Duration of single invocation
📝 - Benchmark Source		 2.07 μs 2.79 μs 1.35 0.02 False
Singleton - Duration of single invocation
📝 - Benchmark Source		 2.14 μs 2.86 μs 1.34 0.01 False
Singleton - Duration of single invocation
📝 - Benchmark Source		 2.15 μs 2.86 μs 1.33 0.02 False
Transient - Duration of single invocation
📝 - Benchmark Source		 2.06 μs 2.81 μs 1.36 0.01 False
Transient - Duration of single invocation
📝 - Benchmark Source		 2.04 μs 3.04 μs 1.49 0.02 False
Scoped - Duration of single invocation
📝 - Benchmark Source		 2.19 μs 3.68 μs 1.68 0.01 False
Singleton - Duration of single invocation
📝 - Benchmark Source		 2.12 μs 2.85 μs 1.34 0.01 False
Scoped - Duration of single invocation
📝 - Benchmark Source		 2.18 μs 4.43 μs 2.03 0.01 False
Scoped - Duration of single invocation
📝 - Benchmark Source		 2.22 μs 3.70 μs 1.67 0.02 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.TimeToFirstService*'

Payloads

Baseline
Compare

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Transient(Mode: "ILEmit")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.8074656687937947 > 2.169533256426224.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -58.719093257450126 (T) = (0 -2792.851037422641) / Math.Sqrt((352.60395059866795 / (25)) + (543.8548409516927 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.3469069695147478 = (2073.529279033151 - 2792.851037422641) / 2073.529279033151 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Scoped(Mode: "Expressions")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.7449060919169086 > 2.313329304203398.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -54.923040510076426 (T) = (0 -3697.0819345898108) / Math.Sqrt((537.825920760676 / (25)) + (2789.87435868882 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.662069476493108 = (2224.384712479341 - 3697.0819345898108) / 2224.384712479341 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "Dynamic")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.85680345824012 > 2.252353329493443.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -31.263725112178555 (T) = (0 -2830.783672344858) / Math.Sqrt((432.5965154995103 / (25)) + (1861.7552338641817 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.3204047668454227 = (2143.875683748007 - 2830.783672344858) / 2143.875683748007 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Transient(Mode: "Dynamic")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.7852518123062144 > 2.1674703674283275.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -39.18403923643714 (T) = (0 -2796.1699528407926) / Math.Sqrt((601.8852429331239 / (25)) + (1258.76112347285 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.347574743169357 = (2074.964648168226 - 2796.1699528407926) / 2074.964648168226 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "Runtime")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.8638615516169157 > 2.236946583419647.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -13.674900818646606 (T) = (0 -2867.3547131966916) / Math.Sqrt((693.2125990057299 / (25)) + (10946.757199621194 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.33467176812110977 = (2148.3594556234775 - 2867.3547131966916) / 2148.3594556234775 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "ILEmit")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.860767285628019 > 2.2581899871910105.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -28.459806100580693 (T) = (0 -2814.6439064723313) / Math.Sqrt((394.51806269517533 / (25)) + (1602.6110112794086 / (3))) is less than -2.055529438636913 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (3) - 2, .025) and -0.3108366661755923 = (2147.211761083968 - 2814.6439064723313) / 2147.211761083968 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Transient(Mode: "Expressions")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.8059250772847997 > 2.156217797528226.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -109.30110293957469 (T) = (0 -2792.969034167149) / Math.Sqrt((582.6300813353151 / (25)) + (82.0607472504304 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.3496327992981993 = (2069.428836954374 - 2792.969034167149) / 2069.428836954374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Transient(Mode: "Runtime")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.039595297918162 > 2.1777183476809574.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -11.301501377442008 (T) = (0 -2841.6026985209705) / Math.Sqrt((474.54920924768743 / (25)) + (18310.227520502194 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.3691923776596196 = (2075.3860048345896 - 2841.6026985209705) / 2075.3860048345896 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Scoped(Mode: "ILEmit")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.676834172199614 > 2.3172891684729664.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -35.680484435747516 (T) = (0 -3670.6950226613326) / Math.Sqrt((1349.1500413067165 / (25)) + (6340.841753427479 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.6489291581903674 = (2226.108383388508 - 3670.6950226613326) / 2226.108383388508 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "Expressions")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.846683093547583 > 2.228866049088209.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -30.937862344587977 (T) = (0 -2832.288570055267) / Math.Sqrt((412.0683729006819 / (25)) + (1919.365963300643 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.321619113764339 = (2143.0444978872324 - 2832.288570055267) / 2143.0444978872324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Scoped(Mode: "Dynamic")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.426027441140356 > 2.3147573900974505.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -9.688500845345093 (T) = (0 -3906.8244227089135) / Math.Sqrt((390.3164674535471 / (25)) + (122453.70127502766 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.7668145123735033 = (2211.2250014635456 - 3906.8244227089135) / 2211.2250014635456 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.DependencyInjection.TimeToFirstService.Scoped(Mode: "Runtime")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.70442340470864 > 2.3019733705636787.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -11.795904307933982 (T) = (0 -3572.4373834456756) / Math.Sqrt((510.8867420414421 / (25)) + (53140.550141296815 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.6151848665431371 = (2211.7823522526583 - 3572.4373834456756) / 2211.7823522526583 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline b9c99648783d0c28afdee394a93516c0bd228165
Compare 9409e2d5019dc46a7a8fd1c59b19a95ddaa3106e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
CreateInstance - Duration of single invocation
📝 - Benchmark Source		 656.05 ns 1.13 μs 1.71 0.20 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark*'

Payloads

Baseline
Compare

Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark.CreateInstance

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1250784875784876 > 713.5991620104128.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -21.584278836659305 (T) = (0 -1139.3568733174607) / Math.Sqrt((1034.8133213404726 / (25)) + (1509.4652706212294 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -0.633100549870702 = (697.6648641797759 - 1139.3568733174607) / 697.6648641797759 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline b9c99648783d0c28afdee394a93516c0bd228165
Compare 9409e2d5019dc46a7a8fd1c59b19a95ddaa3106e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.DependencyInjection.GetServiceIEnumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
Scoped - Duration of single invocation
📝 - Benchmark Source		 31.63 ns 115.12 ns 3.64 0.27 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.GetServiceIEnumerable*'

Payloads

Baseline
Compare

Microsoft.Extensions.DependencyInjection.GetServiceIEnumerable.Scoped

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 115.11620215701704 > 33.50270896524814.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -35.204332932944325 (T) = (0 -118.81883672568196) / Math.Sqrt((4.423021630949348 / (25)) + (23.338142593740454 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -2.655525944638927 = (32.50389643655456 - 118.81883672568196) / 32.50389643655456 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline b9c99648783d0c28afdee394a93516c0bd228165
Compare 9409e2d5019dc46a7a8fd1c59b19a95ddaa3106e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.DependencyInjection.GetService

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
Scoped - Duration of single invocation
📝 - Benchmark Source		 31.57 ns 112.68 ns 3.57 0.19 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.GetService*'

Payloads

Baseline
Compare

Microsoft.Extensions.DependencyInjection.GetService.Scoped

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 112.679349578373 > 33.08173716511241.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/13/2023 3:01:51 PM, 7/17/2023 10:23:26 PM falls between 7/8/2023 10:33:35 PM and 7/17/2023 10:23:26 PM.
IsRegressionStdDev: Marked as regression because -159.52245262501265 (T) = (0 -112.13476230036886) / Math.Sqrt((0.6216975524946721 / (25)) + (0.9190557309194967 / (4))) is less than -2.051830516474954 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (4) - 2, .025) and -2.54430120386857 = (31.638045372096162 - 112.13476230036886) / 31.638045372096162 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: benjaminpetit
Labels:

os-windows, tenet-performance, tenet-performance-benchmarks, arch-x64, untriaged, runtime-coreclr, area-Extensions-DependencyInjection, needs-area-label
Milestone: -

@steveharter
Copy link
Member

Regressions were expected due to adding and using the ServiceIdentifier struct to key on named services. We need to see if the current implementation can be improved, especially around the scoping benchmarks which can be ~2x slower.

@steveharter steveharter removed the untriaged New issue has not been triaged by the area owner label Jul 19, 2023
@steveharter steveharter added this to the 8.0.0 milestone Jul 19, 2023
@ilmax
Copy link

ilmax commented Jul 20, 2023

I think ~2x slower in dependency injection for transient services is too much of price to pay for something fairly uncommon like Keyed Services.
This will regress all customers out there using dependency injection and benefit only few of them that will take advantage of this functionality.
Can we make this feature more pay per play winning back the perf left on the table or else revert this change altogether.
Who wants Keyed Services can always implement tiny workaround to have the same functionality.

@cincuranet
Copy link
Contributor

cincuranet commented Jul 20, 2023 •
edited
Loading

Similar on arm64:

@steveharter
Copy link
Member

We'll need to do some digging here; if it is related to ServiceIdentifier then we can avoid the instantiation of ServiceIdentifier for the default Type cases (i.e. use object instead, and pass the Type).

@vcsjones vcsjones removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jul 26, 2023
@davidfowl
Copy link
Member

cc @adityamandaleeka @benjaminpetit

@benjaminpetit
Copy link
Member

I started to investigate a bit; it seems the main culprit is the attributes parsing in the constructor when trying to get the service for the first time.

Note that it's only the first GetService for a given type that is slower, subsequent calls to GetService don't seems to be slower than before.

It would be interesting to add some case that use the factory method to see if there is any regression.

I managed to get some perf back on my local machine, I will open a PR when ready.

@benjaminpetit benjaminpetit mentioned this issue Aug 3, 2023
Merged
This was referenced Aug 7, 2023
Closed
Closed
@steveharter
Copy link
Member

Closing; there are no more active investigations for v8. cc @benjaminpetit

Note the recent improvements mentioned in in dotnet/perf-autofiling-issues#20567 and dotnet/perf-autofiling-issues#20580 which essentially get the "scoped" cases back to par (not necessarily first-time usage, but many calls to GetService).

@steveharter steveharter mentioned this issue Aug 28, 2023
Closed
@DrewScoggins DrewScoggins mentioned this issue Sep 5, 2023
Closed
@ghost ghost locked as resolved and limited conversation to collaborators Sep 16, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@benjaminpetit benjaminpetit

Labels
arch-x64 area-Extensions-DependencyInjection os-windows runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests
8 participants
@davidfowl @vcsjones @EgorBo @ilmax @cincuranet @steveharter @jeffschwMSFT @benjaminpetit