-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
.NET 7.0 Preview 5 Microbenchmarks Performance Study Report #70590
Comments
Tagging subscribers to this area: @dotnet/area-meta Issue DetailsDataThis time we have covered following configs:
Most of the benchmarks were run on bare-metal machines, some were executed via WSL. This would not be possible without the help from: @AndyAyersMS @carlossanlop @janvorli @jeffhandley @sblom and @tannergooding who contributed their results and time. An addtional thank you to @mrsharm for shadowing the creation of this report! The full report generated by the tool is available here. You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them! Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. Preview 4 report can be found here. Notable Improvements
RegressionsBy design
Investigation in progress
Already solved
Noise, flaky or multimodalThe following benchmarks showed up in the report generated by the tool, but were not actual regressions:
StatisticsTotal: 81617 Statistics per Architecture
Statistics per Operating System
Statistics per Namespace
Big thanks to everyone involved!
|
Am I reading this correctly? |
@TonyValenti That's correct. One of the purposes of this report is to provide an additional line of defense against unintended regressions that can slip past us. We analyze the data and try to track down what might be causing specific regressions. Sometimes, the reported regression ends up just being a noisy benchmark. Sometimes, there is an actual regression, and the report helps notify the right people about the problem, while providing data to help them solve it.
If you happen to be looking at the full report generated by the tool when you come to this conclusion, I want to clarify that the data is sorted from most regressed to most improved. Only looking at the top of the file will lead to that conclusion, but the whole report is 100k lines long, and most benchmarks in the middle end up not changing on a month-to-month basis. Let me know if that cleared things up! |
Also worth noting that some regressions are from a bug fix (#68691). If you factor those out it won't look quite as bad. |
There was also a particularly high number of noisy benchmarks this month, a lot of which were incorrectly flagged as regressions by the tool. This unfortunately skews the overall statistics. You can see specifics in my |
On top of what others said, # tests does not necessarily correlate well with # distinct regressions, # apps affected, etc. often a regression in many tests is caused by one or two regressions, which can then get fixed. Also, this is a point in time during development -- the key thing is the system spots regressions in time to fix them. |
Data
This time we have covered following configs:
Most of the benchmarks were run on bare-metal machines, some were executed via WSL.
This would not be possible without the help from: @AndyAyersMS @carlossanlop @janvorli @jeffhandley @sblom and @tannergooding who contributed their results and time. An addtional thank you to @mrsharm for shadowing the creation of this report!
The full report generated by the tool is available here. You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them!
Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. Preview 4 report can be found here.
Notable Improvements
System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterLE
,System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterBE
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{2,4}(Tom|Sawyer|Huckleberry|Finn)", Options: Compiled)
,System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{0,2}(Tom|Sawyer|Huckleberry|Finn)", Options: Compiled)
PerfLabTests.CastingPerf.IFooFooIsIFoo
,PerfLabTests.CastingPerf.ObjFooIsObj2
,PerfLabTests.CastingPerf2.CastingPerf.IFooFooIsIFoo
System.Memory.Span<Byte>.Reverse(Size: 512)
,System.Memory.Span<Char>.Reverse(Size: 512)
,System.Memory.Span<Int32>.Reverse(Size: 512)
,System.Tests.Perf_Array.Reverse
System.Memory.Span<Byte>.Fill(Size: 512)
,System.Memory.Span<Byte>.Clear(Size: 512)
Regressions
By design
System.Numerics.Tests.Perf_Quaternion.EqualsBenchmark
,System.Numerics.Tests.Perf_Vector3.EqualsBenchmark
,System.Numerics.Tests.Perf_Vector2.EqualsBenchmark
,System.Numerics.Tests.Perf_Matrix3x2.EqualsBenchmark
,System.Numerics.Tests.Perf_Vector4.EqualsBenchmark
Investigation in progress
System.Text.Perf_Utf8Encoding.GetString(Input: Chinese)
,System.Text.Perf_Utf8Encoding.GetString(Input: Greek)
,System.Text.Perf_Utf8Encoding.GetString(Input: Cyrillic)
,System.Text.Perf_Utf8Encoding.GetBytes(Input: Chinese)
char.Is*
update #69727char.Is*
update #69727 (comment)System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, None, False))
,System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, IgnoreCase, False))
,System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (, None, False))
,System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, OrdinalIgnoreCase, False))
,System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, IgnoreNonSpace, False))
,System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (, IgnoreCase, False))
System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en-US, OrdinalIgnoreCase))
System.Tests.Perf_String.Format_MultipleArgs
System.Collections.Sort<IntStruct>.List(Size: 512)
,System.Collections.Sort<IntStruct>.Array(Size: 512)
System.Memory.Span<Byte>.Reverse(Size: 512)
,System.Memory.Span<Char>.Reverse(Size: 512)
,System.Memory.Span<Int32>.Reverse(Size: 512)
,System.Tests.Perf_Array.Reverse
Already solved
MicroBenchmarks.Serializers.Json_FromStream<LoginViewModel>.DataContractJsonSerializer_
,MicroBenchmarks.Serializers.Json_FromStream<Location>.DataContractJsonSerializer_
,MicroBenchmarks.Serializers.Xml_FromStream<SimpleStructWithProperties>.DataContractSerializer_
,MicroBenchmarks.Serializers.Xml_FromStream<LoginViewModel>.DataContractSerializer_
,MicroBenchmarks.Serializers.Xml_FromStream<CollectionsOfPrimitives>.DataContractSerializer_
, MicroBenchmarks.Serializers.Xml_FromStream.DataContractSerializer_`Benchstone.BenchI.IniArray.Test
Noise, flaky or multimodal
The following benchmarks showed up in the report generated by the tool, but were not actual regressions:
System.Text.RegularExpressions.Tests.*.Count(*, Options: None)
Options: None
regex tests, but historical data indicates it is probably noise. These are worth keeping an eye on.BenchmarksGame.RegexRedux_5.RunBench(options: None)
System.Threading.Channels.Tests.UnboundedChannelPerfTests.PingPong
System.Tests.Perf_Char.Char_ToLowerInvariant(input: "Hello World!")
,System.Tests.Perf_Char.Char_ToUpperInvariant(input: "Hello World!")
System.Buffers.Tests.RentReturnArrayPoolTests<Object>.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)
System.Net.Security.Tests.SslStreamTests.ConcurrentReadWrite
System.Numerics.Tests.Perf_Matrix4x4.IdentityBenchmark
PerfLabTests.CastingPerf.CheckListIsVariantGenericInterface
System.Buffers.Tests.RentReturnArrayPoolTests<Byte>.MultipleSerial(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: False)
System.Numerics.Tests.Perf_Vector3.DistanceSquaredBenchmark
System.Numerics.Tests.Perf_Vector2.ReflectBenchmark
System.Numerics.Tests.Perf_VectorOf<Double>.NegateBenchmark
System.Numerics.Tests.Perf_VectorOf<Int32>.XorBenchmark
PerfLabTests.LowLevelPerf.GenericGenericMethod
PerfLabTests.LowLevelPerf.GenericClassWithSTringGenericInstanceMethod
System.Memory.Span<Int32>.IndexOfAnyFourValues(Size: 512)
System.Collections.IterateForEach<Int32>.LinkedList(Size: 512)
System.Memory.Span<Byte>.SequenceCompareTo(Size: 512)
Statistics
Total: 81617
Same: 79.67 %
Slower: 7.73 %
Faster: 4.27 %
Noise: 8.27 %
Unknown: 0.05 %
Statistics per Architecture
Statistics per Operating System
Statistics per Namespace
Big thanks to everyone involved!
The text was updated successfully, but these errors were encountered: