Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark #94301

Closed
performanceautofiler bot opened this issue Oct 31, 2023 · 14 comments
Closed

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark #94301

performanceautofiler bot opened this issue Oct 31, 2023 · 14 comments
Labels
arch-x64 area-System.Runtime needs-further-triage Issue has been initially triaged, but needs deeper consideration or reconsideration os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Oct 31, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 40f23e33881df2ff1c3451bbb28d5c3dce7ecf68
Compare 323eef2f6158fe7dee9b2773b4e33e44ea9bc70b
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
8.43 ns 9.54 ns 1.13 0.08 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*'

Payloads

Baseline
Compare

Microsoft.Extensions.Primitives.StringSegmentBenchmark.GetSegmentHashCode

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 40f23e33881df2ff1c3451bbb28d5c3dce7ecf68
Compare 323eef2f6158fe7dee9b2773b4e33e44ea9bc70b
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Net.Primitives.Tests.CredentialCacheTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
112.05 ns 118.87 ns 1.06 0.19 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Primitives.Tests.CredentialCacheTests*'

Payloads

Baseline
Compare

System.Net.Primitives.Tests.CredentialCacheTests.ForEach(uriCount: 0, hostPortCount: 10)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-x64 os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels Oct 31, 2023
@performanceautofiler performanceautofiler bot mentioned this issue Oct 31, 2023
Closed
@cincuranet
Copy link
Contributor

The diff for GetSegmentHashCode is 56121e4...dd89eb7. Rest looks like noise.

@cincuranet cincuranet removed the untriaged New issue has not been triaged by the area owner label Nov 2, 2023
@cincuranet cincuranet transferred this issue from dotnet/perf-autofiling-issues Nov 2, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Nov 2, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Nov 2, 2023
@cincuranet cincuranet changed the title [Perf] Linux/x64: 2 Regressions on 10/26/2023 8:29:49 PM Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark Nov 2, 2023
@cincuranet cincuranet added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Nov 2, 2023
@ghost
Copy link

ghost commented Nov 7, 2023

Tagging subscribers to this area: @dotnet/area-extensions-primitives
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 40f23e33881df2ff1c3451bbb28d5c3dce7ecf68
Compare 323eef2f6158fe7dee9b2773b4e33e44ea9bc70b
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
8.43 ns 9.54 ns 1.13 0.08 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*'

Payloads

Baseline
Compare

Microsoft.Extensions.Primitives.StringSegmentBenchmark.GetSegmentHashCode

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 40f23e33881df2ff1c3451bbb28d5c3dce7ecf68
Compare 323eef2f6158fe7dee9b2773b4e33e44ea9bc70b
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Net.Primitives.Tests.CredentialCacheTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
112.05 ns 118.87 ns 1.06 0.19 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Primitives.Tests.CredentialCacheTests*'

Payloads

Baseline
Compare

System.Net.Primitives.Tests.CredentialCacheTests.ForEach(uriCount: 0, hostPortCount: 10)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

os-linux, tenet-performance, tenet-performance-benchmarks, arch-x64, untriaged, runtime-coreclr, area-Extensions-Primitives, needs-area-label
Milestone: -

@tarekgh tarekgh added area-System.Runtime and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners area-Extensions-Primitives labels Nov 7, 2023
@ghost
Copy link

ghost commented Nov 7, 2023

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 40f23e33881df2ff1c3451bbb28d5c3dce7ecf68
Compare 323eef2f6158fe7dee9b2773b4e33e44ea9bc70b
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
8.43 ns 9.54 ns 1.13 0.08 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*'

Payloads

Baseline
Compare

Microsoft.Extensions.Primitives.StringSegmentBenchmark.GetSegmentHashCode

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 40f23e33881df2ff1c3451bbb28d5c3dce7ecf68
Compare 323eef2f6158fe7dee9b2773b4e33e44ea9bc70b
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Net.Primitives.Tests.CredentialCacheTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
112.05 ns 118.87 ns 1.06 0.19 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Primitives.Tests.CredentialCacheTests*'

Payloads

Baseline
Compare

System.Net.Primitives.Tests.CredentialCacheTests.ForEach(uriCount: 0, hostPortCount: 10)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

area-System.Runtime, os-linux, tenet-performance, tenet-performance-benchmarks, arch-x64, untriaged, runtime-coreclr
Milestone: -

@tarekgh
Copy link
Member

tarekgh commented Nov 7, 2023

We didn't touch Primitives to cause any regression. GetSegmentHashCode also is just calling string.GetHashCode. The closest change maybe related is #93987. @thaystg could this be related to your change?

@thaystg
Copy link
Member

thaystg commented Nov 7, 2023

Maybe yes, I merged a fixed for interpreter yesterday. Probably it's fixed.

@cincuranet
Copy link
Contributor

cincuranet commented Nov 7, 2023 •
edited
Loading

So far the regression is still there (last results from Tue Nov 07 2023 10:04:49 GMT+0000):
image

@thaystg
Copy link
Member

thaystg commented Nov 7, 2023 •
edited
Loading

I don't think this is related I only touched on sources related to mono runtime, this regression is not using mono runtime.

image

@tarekgh
Copy link
Member

tarekgh commented Nov 7, 2023

Going through all the commits I can see anyone can cause this regression.

@elinor-fung could the corerun change 62191f8 be related?

@tarekgh
Copy link
Member

tarekgh commented Nov 7, 2023 •
edited
Loading

By the way this issue is also listing a regression in System.Net.Primitives.Tests.CredentialCacheTests which should be different than Microsoft.Extensions.Primitives. I don't know why the issue title changed to include this one only.

@cincuranet
Copy link
Contributor

System.Net.Primitives.Tests.CredentialCacheTests is not a real regression (after human triaging). We change the title in triaging if there's only one set of regressions and the rest is unrelated.

@tarekgh
Copy link
Member

tarekgh commented Nov 7, 2023

@cincuranet can you elaborate more about that. I am seeing System.NET regression is 6%. I know the extensions regression looks big 13% but reality it shows 1.11 ns. while system.net shows 6.82 ns.

@elinor-fung
Copy link
Member

could the corerun change 62191f8 be related?

I wouldn't imagine so. That change just removed an extra / in the paths for the TPA.

@cincuranet
Copy link
Contributor

@tarekgh The System.Net.Primitives.Tests.CredentialCacheTests is noisy/less stable. These are latest results. Basically same as "before".

image

@tannergooding tannergooding added the needs-further-triage Issue has been initially triaged, but needs deeper consideration or reconsideration label Jun 24, 2024
@tannergooding tannergooding removed the untriaged New issue has not been triaged by the area owner label Jun 24, 2024
@stephentoub
Copy link
Member

The tests in question are either bimodal or have recovered.

@stephentoub stephentoub closed this as not planned Won't fix, can't repro, duplicate, stale Jul 22, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Aug 22, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-x64 area-System.Runtime needs-further-triage Issue has been initially triaged, but needs deeper consideration or reconsideration os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@stephentoub @thaystg @cincuranet @jeffschwMSFT @tannergooding @tarekgh @elinor-fung