[Perf -152%] System.IO.Tests.Perf_Path.GetRandomFileName

[DrewScoggins]

Run Information

Architecture	x64
OS	ubuntu 18.04
Changes	diff

Regressions in System.IO.Tests.Perf_Path

Benchmark	Baseline	Test	Test/Base	Modality	Baseline Outlier
GetRandomFileName	695.30 ns	1.76 μs	2.54		False

Related Issue on x86 Windows

[Perf -15%] System.IO.Tests.Perf_Path.GetPathRoot

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.IO.Tests.Perf_Path*';

Histogram

System.IO.Tests.Perf_Path.GetRandomFileName

[ 607.180 ;  641.303) | @@@
[ 641.303 ;  706.017) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[ 706.017 ;  763.521) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[ 763.521 ;  828.235) | 
[ 828.235 ;  892.949) | 
[ 892.949 ;  957.663) | 
[ 957.663 ; 1022.377) | 
[1022.377 ; 1087.091) | 
[1087.091 ; 1151.805) | 
[1151.805 ; 1216.519) | 
[1216.519 ; 1281.233) | 
[1281.233 ; 1345.947) | 
[1345.947 ; 1410.660) | 
[1410.660 ; 1475.374) | 
[1475.374 ; 1540.088) | 
[1540.088 ; 1604.802) | 
[1604.802 ; 1669.516) | 
[1669.516 ; 1741.583) | 
[1741.583 ; 1828.164) | @@
[1828.164 ; 1893.916) | @@

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[danmoseley]

@DrewScoggins the diff link does not work. Can you please share the diff bracketing this jump?

[DrewScoggins]

Yeah, there seems to be a bug here if the points you are trying to diff go across the switch to the runtime repo. You can use the report to generate a tight diff, again as long as the comparison you are trying to do is after the switch to the runtime repo. To do this

1. Click on this link Historical Data in Reporting System
2. Find the point before the regression and click on that point
   
3. Click the Set Baseline button then close the flyout by clicking on the red X.
4. Find the point after the regression and click on that point
   
5. Click Set Compare and then you will see links that should lead to a comparison of the changes in the runtime repo.
   

By following these steps I generated this link 4a4e347...d0889f1, which should be able to give you the bounds that you want. Sorry again for the bug in the diff link, I will fix that to link to the installer repo which is what we are actually generating the hashes for, but hopefully this helps you track down diffs going forward 😄

[danmoseley]

Thanks for the info.

will fix that to link to the installer repo which is what we are actually generating the hashes for,

I don't think that's the diff we want for the microbenchmarks. The issues sounds have diff across runtime repo commits surely?

[stephentoub]

I can't repro this locally (on Ubuntu 18.04). I've run the benchmark multiple times, and each time I get results like this:

dotnet run -c Release -f netcoreapp3.1 --runtimes netcoreapp3.1 netcoreapp5.0 --join --filter *System.IO.Tests.Perf_Path.GetRandomFileName*

Method	Job	Runtime	Toolchain	Mean	Error	StdDev	Median	Min	Max	Ratio	Gen 0	Gen 1	Gen 2	Allocated
GetRandomFileName	Job-CSKPPR	.NET Core 3.1	netcoreapp3.1	756.1 ns	3.65 ns	3.41 ns	757.6 ns	747.9 ns	758.6 ns	1.00	0.0061	-	-	48 B
GetRandomFileName	Job-APGSLT	.NET Core 5.0	netcoreapp5.0	744.8 ns	1.59 ns	1.41 ns	745.0 ns	741.3 ns	747.1 ns	0.99	0.0060	-	-	48 B

[jkotas]

Duplicate of #39114 . Both NewGuid and GetRandomFileName use the same underlying OS random number generator.