Added numeric ranking Performance Tests #888

Anipik · 2018-09-11T21:20:02Z

Added benchmarking performance tests for Numeric ranking.

cc @justinormont @sfilipi @danmosemsft @eerhardt @shauheen

Anipik · 2018-09-11T21:41:10Z

Ranking.TrainTest_Multiclass_MSLRWeb10K_Ranking_FastTree: Job-ZGJLIB(Toolchain=netcoreapp2.1, MaxIterationCount=20, WarmupCount=1)
Mean = 72.7866 s, StdErr = 0.5849 s (0.80%); N = 20, StdDev = 2.6156 s
Min = 68.6174 s, Q1 = 70.0802 s, Median = 73.3494 s, Q3 = 74.9717 s, Max = 77.5199 s
IQR = 4.8915 s, LowerFence = 62.7429 s, UpperFence = 82.3089 s
ConfidenceInterval = [70.5153 s; 75.0579 s] (CI 99.9%), Margin = 2.2713 s (3.12% of Mean)
Skewness = -0.09, Kurtosis = 1.66, MValue = 3.14
-------------------- Histogram --------------------
[68.616 s ; 70.533 s) | @@@@@@
[70.533 s ; 72.219 s) | @@
[72.219 s ; 74.846 s) | @@@@@@@
[74.846 s ; 76.379 s) | @@@@
[76.379 s ; 78.363 s) | @
---------------------------------------------------

Ranking.TrainTest_Multiclass_MSLRWeb10K_Ranking_LightGBM: Job-ZGJLIB(Toolchain=netcoreapp2.1, MaxIterationCount=20, WarmupCount=1)
Mean = 65.4654 s, StdErr = 0.9955 s (1.52%); N = 20, StdDev = 4.4522 s
Min = 58.3025 s, Q1 = 62.0859 s, Median = 65.7175 s, Q3 = 68.5847 s, Max = 75.8618 s
IQR = 6.4988 s, LowerFence = 52.3377 s, UpperFence = 78.3329 s
ConfidenceInterval = [61.5993 s; 69.3315 s] (CI 99.9%), Margin = 3.8661 s (5.91% of Mean)
Skewness = 0.1, Kurtosis = 2.64, MValue = 3.25
-------------------- Histogram --------------------
[58.224 s ; 61.721 s) | @@@@@
[61.721 s ; 64.643 s) | @
[64.643 s ; 67.513 s) | @@@@@@@@
[67.513 s ; 70.412 s) | @@@@@
[70.412 s ; 74.427 s) |
[74.427 s ; 77.297 s) | @
---------------------------------------------------

Ranking.Test_Multiclass_MSLRWeb10K_Ranking_FastTree: Job-ZGJLIB(Toolchain=netcoreapp2.1, MaxIterationCount=20, WarmupCount=1)
Mean = 4.5715 s, StdErr = 0.0079 s (0.17%); N = 14, StdDev = 0.0295 s
Min = 4.5317 s, Q1 = 4.5526 s, Median = 4.5653 s, Q3 = 4.5778 s, Max = 4.6321 s
IQR = 0.0252 s, LowerFence = 4.5147 s, UpperFence = 4.6157 s
ConfidenceInterval = [4.5382 s; 4.6047 s] (CI 99.9%), Margin = 0.0333 s (0.73% of Mean)
Skewness = 0.67, Kurtosis = 2.39, MValue = 2
-------------------- Histogram --------------------
[4.524 s ; 4.643 s) | @@@@@@@@@@@@@@
---------------------------------------------------

Toolchain=netcoreapp2.1  MaxIterationCount=20  WarmupCount=1

Method	Mean	Error	StdDev	Extra Metric	Gen 0	Gen 1	Gen 2	Allocated
TrainTest_Multiclass_MSLRWeb10K_Ranking_FastTree	72.787 s	2.2713 s	2.6156 s	-	6247000.0000	1058000.0000	286000.0000	26171.31 MB
TrainTest_Multiclass_MSLRWeb10K_Ranking_LightGBM	65.465 s	3.8661 s	4.4522 s	-	3595000.0000	1684000.0000	267000.0000	304.59 MB
Test_Multiclass_MSLRWeb10K_Ranking_FastTree	4.571 s	0.0333 s	0.0295 s	-	558000.0000	279000.0000	1000.0000	11.93 MB

build.proj

eerhardt · 2018-09-12T16:01:35Z

build.proj

+  </ItemGroup>
+
+  <ItemGroup Condition="'$(IncludeBenchmarkData)' == 'true'" >    
+    <TestFile Include="$(MSBuildThisFileDirectory)/test/data/external/WikiDetoxAnnotated160kRows.tsv"


The duplication here could be simplified using MSBuild. Something along the lines of:

<ItemGroup> <TlcResourceFile Include="WikiDetoxAnnotated160kRows.tsv" /> <TlcResourceFile Include="MSLRWeb10KTrain3.6MRows.tsv" /> <TlcResourceFile Include="MSLRWeb10KValidate1.2MRows.tsv" /> <TlcResourceFile Include="MSLRWeb10KTest1.2MRows.tsv" /> <TlcResourceFile Update="@(TlcResourceFile)"> <Url>http://aka.ms/tlc-resources/benchmarks/%(Identity)</Url> <DestinationFile>$(MSBuildThisFileDirectory)test/data/external/%(Identity)</DestinationFile> </TlcResourceFile> <TestFile Include="@(TlcResourceFile->'$(MSBuildThisFileDirectory)/test/data/external/%(Identity)')" /> </ItemGroup>

I'm not 100% sure it is better, but it reduces the number of times these URLs need to be copied.

That's batching, I think it would only work within a <Target> ?

Yep dan is right. its not working in this case. any other suggestion here ?

It works, you just need to have the right syntax. I've updated the above with actual MSBuild code that works.

Okay thanks :)

test/Microsoft.ML.Benchmarks/Numeric/Ranking.cs

Anipik · 2018-09-12T19:27:49Z

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=2.1.400
  [Host]     : .NET Core 2.1.2 (CoreCLR 4.6.26628.05, CoreFX 4.6.26629.01), 64bit RyuJIT
  Job-QFXMOR : .NET Core 2.1.2 (CoreCLR 4.6.26628.05, CoreFX 4.6.26629.01), 64bit RyuJIT

Toolchain=netcoreapp2.1  MaxIterationCount=20  WarmupCount=1

Method	Mean	Error	StdDev	Extra Metric	Gen 0	Gen 1	Gen 2	Allocated
TrainTest_Multiclass_MSLRWeb10K_Ranking_FastTree	32.993 s	0.5025 s	0.4455 s	-	2762000.0000	192000.0000	56000.0000	15435.34 MB
TrainTest_Multiclass_MSLRWeb10K_Ranking_LightGBM	31.045 s	2.1895 s	2.5215 s	-	1198000.0000	560000.0000	84000.0000	246.64 MB
Test_Multiclass_MSLRWeb10K_Ranking_FastTree	1.153 s	0.0943 s	0.1086 s	-	122000.0000	55000.0000	12000.0000	2.93 MB

build.proj

test/Microsoft.ML.Benchmarks/Numeric/Ranking.cs

sfilipi · 2018-09-13T18:13:05Z

test/Microsoft.ML.Benchmarks/Microsoft.ML.Benchmarks.csproj

-                  Include="..\data\external\WikiDetoxAnnotated160kRows.tsv"
-                  Link="external\WikiDetoxAnnotated160kRows.tsv">
+
+    <TlcResourceFile Update="@(TlcResourceFile)">


TlcResourceFile [](start = 31, length = 15)

call it something non TLC

Agreed, but the reason I chose the name originally is because of the URL:

http://aka.ms/tlc-resources

Can we change this URL? Or make a new aka.ms URL pointing to the same location with a different name?

sfilipi

justinormont

LGTM.

Let's wait to merge until the MSLR-WEB10K dataset is available in the CDN.

test/Microsoft.ML.Benchmarks/Microsoft.ML.Benchmarks.csproj

eerhardt

justinormont · 2018-09-13T21:53:43Z

Please also add the citation to the MSLR-WEB10K dataset.

build.proj

justinormont · 2018-09-14T02:54:55Z

test/Microsoft.ML.Benchmarks/Numeric/Ranking.cs

@@ -11,6 +11,7 @@

 namespace Microsoft.ML.Benchmarks
 {
+    [WarmupCount(8)] // It helps to reduce the standard deviation of these tests.


The normal user is unlikely to pre-train a model 8 times before training their model. This will be representative of the steady state reached when a model is retrained many times, but not very representative of the normal user's interaction w/ ML.net.

Do we know what's causing the time difference between the first run and the later runs? The first run is most representative of what a normal user will experience.

@adamsitnik can you give us a better view here ? Increasing the warmup iterations leads to reducing the standard deviation here

@justinormont do u want me to reduce it ?

cc @danmosemsft

can you give us a better view here ?

@Anipik Unfortunately, it's not that simple. To find out why given benchmark behaves differently for different warmup counts we would have to profile it. It could be that OS gets warmed up and reading the input files becomes faster or anything like that.

The normal user is unlikely to pre-train a model 8 times before training their model.

@justinormont I agree. In that case we should set the WarmupCount to 0, IterationCount to 1 and LaunchCount to 20. Which means that BenchmarkDotNet is going to start a new process 20 times and each time execute the benchmark only once, without any warmup (the real use case) and just exit the process.

Edit: we should most probably have two configs: one for training benchmarks (no warmups) and one for prediction benchmarks (the one we have today)

I will revert warmupCount to 1 for this PR to get merged, we can later follow up with 2 config files as adam suggested

test/data/README.md

justinormont

Looks good, though there's a couple minor things:

Check resource download location:
Added numeric ranking Performance Tests #888 (comment)
For security, use https:
Added numeric ranking Performance Tests #888 (comment)
Reduce extra newlines in the citation: (very minor)
Added numeric ranking Performance Tests #888 (comment)

test/data/README.md

Anipik · 2018-09-17T23:31:55Z

@justinormont cam you take a look here ?

test/data/README.md

Anipik · 2018-09-18T17:08:03Z

@justinormont can you merge this ? I don't have the write access to the repo ?

justinormont · 2018-09-18T17:17:28Z

I'm going to close & re-open this pull request to notify the CI to re-check this PR.

justinormont · 2018-09-18T17:33:54Z

The CI test are failing. Though GitHub says 'in progress', it will say failed soon.

The winequality-white.csv file is the cause.

Error:

2018-09-18T17:27:13.4451915Z  System.IO.IOException : Could not find file 'D:\a\1\s\test\data\external\winequality-white.csv'
2018-09-18T17:27:13.4452129Z Stack Trace:
2018-09-18T17:27:13.4452368Z    at Microsoft.ML.Runtime.Data.MultiFileSource..ctor(String path) in D:\a\1\s\src\Microsoft.ML.Data\DataLoadSave\MultiFileSource.cs:line 31
2018-09-18T17:27:13.4452691Z    at Microsoft.ML.StaticPipelineTesting.Training.SdcaRegression() in D:\a\1\s\test\Microsoft.ML.StaticPipelineTesting\Training.cs:line 29
2018-09-18T17:27:13.4452938Z 
2018-09-18T17:27:13.4453436Z Results File: D:\a\1\s\bin/AnyCPU.Release\Microsoft.ML.StaticPipelineTesting\VssAdministrator_factoryvm-az385_2018-09-18_17_27_12.trx
2018-09-18T17:27:13.4454016Z 
2018-09-18T17:27:13.4454366Z Total tests: 13. Passed: 9. Failed: 4. Skipped: 0.

Anipik · 2018-09-18T17:41:11Z

Looking into it

eerhardt · 2018-09-18T17:42:43Z

It's a well-known issue that the wine dataset isn't working right now. @artidoro is working on it.

justinormont · 2018-09-18T17:52:14Z

Related issue for the Wine dataset: #889 Hot linking to a UCI dataset

Currently, the UCI web server is non-responsive, causing the dataset to not download, and the test to fail.

Anipik · 2018-09-19T00:18:22Z

@justinormont the ci is green, can we go ahead and merge this one ?

justinormont · 2018-09-19T04:32:22Z

@Anipik, the merge is waiting on a merge conflict, can you look in to it?

Anipik · 2018-09-19T04:40:27Z

@justinormont i resolved the conflict

Missing semicolon is causing the build the fail: ``` 2018-09-19T04:41:39.2181028Z Datasets.cs(171,10): error CS1002: ; expected [/__w/3/s/test/Microsoft.ML.TestFramework/Microsoft.ML.TestFramework.csproj] 2018-09-19T04:41:39.7812509Z Microsoft.ML.StandardLearners -> /__w/3/s/bin/AnyCPU.Debug/Microsoft.ML.StandardLearners/netstandard2.0/Microsoft.ML.StandardLearners.dll 2018-09-19T04:41:40.7120753Z Microsoft.ML.HalLearners -> /__w/3/s/bin/AnyCPU.Debug/Microsoft.ML.HalLearners/netstandard2.0/Microsoft.ML.HalLearners.dll 2018-09-19T04:41:40.8804119Z Microsoft.ML.Ensemble -> /__w/3/s/bin/AnyCPU.Debug/Microsoft.ML.Ensemble/netstandard2.0/Microsoft.ML.Ensemble.dll 2018-09-19T04:41:40.9555420Z Microsoft.ML.LightGBM -> /__w/3/s/bin/AnyCPU.Debug/Microsoft.ML.LightGBM/netstandard2.0/Microsoft.ML.LightGBM.dll 2018-09-19T04:41:41.5610322Z Microsoft.ML.PipelineInference -> /__w/3/s/bin/AnyCPU.Debug/Microsoft.ML.PipelineInference/netstandard2.0/Microsoft.ML.PipelineInference.dll 2018-09-19T04:41:42.4887819Z Microsoft.ML.Console -> /__w/3/s/bin/AnyCPU.Debug/Microsoft.ML.Console/netcoreapp2.0/MML.dll 2018-09-19T04:41:45.7637388Z Microsoft.ML.FSharp.Tests -> /__w/3/s/bin/AnyCPU.Debug/Microsoft.ML.FSharp.Tests/netcoreapp2.1/Microsoft.ML.FSharp.Tests.dll 2018-09-19T04:41:45.7926386Z /__w/3/s/dir.traversal.targets(25,5): error : Build failed. See earlier errors. [/__w/3/s/build.proj] 2018-09-19T04:41:45.8133725Z 2018-09-19T04:41:45.8152732Z Build FAILED. ```

justinormont · 2018-09-19T20:25:37Z

Thanks @Anipik for all the unexpected work needed in this pull request.

eerhardt reviewed Sep 12, 2018

View reviewed changes

build.proj Outdated Show resolved Hide resolved

eerhardt reviewed Sep 12, 2018

View reviewed changes

test/Microsoft.ML.Benchmarks/Numeric/Ranking.cs Outdated Show resolved Hide resolved

eerhardt reviewed Sep 12, 2018

View reviewed changes

test/Microsoft.ML.Benchmarks/Numeric/Ranking.cs Outdated Show resolved Hide resolved

eerhardt reviewed Sep 12, 2018

View reviewed changes

test/Microsoft.ML.Benchmarks/Numeric/Ranking.cs Outdated Show resolved Hide resolved

justinormont self-requested a review September 12, 2018 19:35

justinormont mentioned this pull request Sep 12, 2018

Perf benchmarks for optimization #711

Closed

justinormont reviewed Sep 12, 2018

View reviewed changes

build.proj Outdated Show resolved Hide resolved

justinormont reviewed Sep 12, 2018

View reviewed changes

test/Microsoft.ML.Benchmarks/Numeric/Ranking.cs Outdated Show resolved Hide resolved

sfilipi reviewed Sep 13, 2018

View reviewed changes

sfilipi approved these changes Sep 13, 2018

View reviewed changes

justinormont approved these changes Sep 13, 2018

View reviewed changes

eerhardt reviewed Sep 13, 2018

View reviewed changes

test/Microsoft.ML.Benchmarks/Microsoft.ML.Benchmarks.csproj Outdated Show resolved Hide resolved

eerhardt reviewed Sep 13, 2018

View reviewed changes

test/Microsoft.ML.Benchmarks/Microsoft.ML.Benchmarks.csproj Outdated Show resolved Hide resolved

eerhardt approved these changes Sep 13, 2018

View reviewed changes

justinormont reviewed Sep 14, 2018

View reviewed changes

build.proj Outdated Show resolved Hide resolved

justinormont reviewed Sep 14, 2018

View reviewed changes

test/data/README.md Outdated Show resolved Hide resolved

justinormont reviewed Sep 14, 2018

View reviewed changes

test/data/README.md Outdated Show resolved Hide resolved

justinormont suggested changes Sep 14, 2018

View reviewed changes

eerhardt reviewed Sep 14, 2018

View reviewed changes

test/data/README.md Outdated Show resolved Hide resolved

justinormont reviewed Sep 14, 2018

View reviewed changes

test/data/README.md Outdated Show resolved Hide resolved

Anipik added 5 commits September 17, 2018 15:32

added numeric ranking tests

ab5d473

feedback, indentation added

1eb2b1f

names change, url changes and warmcount changes

6ff4680

url corrected, https corrected and extra lines removed

599f719

tlc changed to console environment

fe01702

justinormont reviewed Sep 18, 2018

View reviewed changes

test/data/README.md Outdated Show resolved Hide resolved

justinormont approved these changes Sep 18, 2018

View reviewed changes

justinormont reviewed Sep 18, 2018

View reviewed changes

test/data/README.md Outdated Show resolved Hide resolved

Anipik and others added 2 commits September 17, 2018 22:19

https and closing brace added

638d0c8

Code block for MSLR-WEB10K/MSLR-WEB30K to format citation

2616671

justinormont closed this Sep 18, 2018

justinormont reopened this Sep 18, 2018

Merge branch 'master' into NumericRanking

4508c52

Merge branch 'master' into NumericRanking

59de38a

justinormont merged commit 86f4d93 into dotnet:master Sep 19, 2018

Anipik deleted the NumericRanking branch October 10, 2018 18:22

ghost locked as resolved and limited conversation to collaborators Mar 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added numeric ranking Performance Tests #888

Added numeric ranking Performance Tests #888

Anipik commented Sep 11, 2018

Anipik commented Sep 11, 2018

eerhardt Sep 12, 2018 •

edited

Loading

danmoseley Sep 12, 2018

Anipik Sep 12, 2018

eerhardt Sep 12, 2018

Anipik Sep 12, 2018

Anipik commented Sep 12, 2018

sfilipi Sep 13, 2018

eerhardt Sep 13, 2018

sfilipi left a comment

justinormont left a comment

eerhardt left a comment

justinormont commented Sep 13, 2018

justinormont Sep 14, 2018 •

edited

Loading

Anipik Sep 14, 2018

Anipik Sep 14, 2018

adamsitnik Sep 17, 2018 •

edited

Loading

Anipik Sep 17, 2018 •

edited

Loading

justinormont left a comment •

edited

Loading

Anipik commented Sep 17, 2018

Anipik commented Sep 18, 2018

justinormont commented Sep 18, 2018

justinormont commented Sep 18, 2018 •

edited

Loading

Anipik commented Sep 18, 2018

eerhardt commented Sep 18, 2018

justinormont commented Sep 18, 2018

Anipik commented Sep 19, 2018

justinormont commented Sep 19, 2018

Anipik commented Sep 19, 2018

justinormont commented Sep 19, 2018

Added numeric ranking Performance Tests #888

Added numeric ranking Performance Tests #888

Conversation

Anipik commented Sep 11, 2018

Anipik commented Sep 11, 2018

eerhardt Sep 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Anipik commented Sep 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi left a comment

Choose a reason for hiding this comment

justinormont left a comment

Choose a reason for hiding this comment

eerhardt left a comment

Choose a reason for hiding this comment

justinormont commented Sep 13, 2018

justinormont Sep 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamsitnik Sep 17, 2018 • edited Loading

Choose a reason for hiding this comment

Anipik Sep 17, 2018 • edited Loading

Choose a reason for hiding this comment

justinormont left a comment • edited Loading

Choose a reason for hiding this comment

Anipik commented Sep 17, 2018

Anipik commented Sep 18, 2018

justinormont commented Sep 18, 2018

justinormont commented Sep 18, 2018 • edited Loading

Anipik commented Sep 18, 2018

eerhardt commented Sep 18, 2018

justinormont commented Sep 18, 2018

Anipik commented Sep 19, 2018

justinormont commented Sep 19, 2018

Anipik commented Sep 19, 2018

justinormont commented Sep 19, 2018

eerhardt Sep 12, 2018 •

edited

Loading

justinormont Sep 14, 2018 •

edited

Loading

adamsitnik Sep 17, 2018 •

edited

Loading

Anipik Sep 17, 2018 •

edited

Loading

justinormont left a comment •

edited

Loading

justinormont commented Sep 18, 2018 •

edited

Loading