Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speedup InceptionV3 and GoogleNet tests on windows (upgrade scipy) #3196

Conversation

datumbox
Copy link
Contributor

@datumbox datumbox commented Dec 19, 2020

Rough improvement: ~96%

327.29s call     test/test_models.py::ModelTester::test_inception_v3_cpu
326.23s call     test/test_models.py::ModelTester::test_inception_v3_cuda
135.34s call     test/test_models.py::ModelTester::test_googlenet_cuda
133.91s call     test/test_models.py::ModelTester::test_googlenet_cpu
5.43s call     test/test_models.py::ModelTester::test_inceptionv3_eval
3.04s call     test/test_models.py::ModelTester::test_googlenet_eval
7.97s call     test/test_models.py::ModelTester::test_googlenet_cuda
7.41s call     test/test_models.py::ModelTester::test_inception_v3_cpu
5.24s call     test/test_models.py::ModelTester::test_inception_v3_cuda
4.52s call     test/test_models.py::ModelTester::test_googlenet_cpu
4.50s call     test/test_models.py::ModelTester::test_inception_v3_eval
2.74s call     test/test_models.py::ModelTester::test_googlenet_eval

The above are rough estimations based on a couple or runs.

The problem is traced to the use of scipy in the weight initialization of the two models:

import scipy.stats as stats
stddev = m.stddev if hasattr(m, 'stddev') else 0.1
X = stats.truncnorm(-2, 2, scale=stddev)

import scipy.stats as stats
X = stats.truncnorm(-2, 2, scale=0.01)
values = torch.as_tensor(X.rvs(m.weight.numel()), dtype=m.weight.dtype)

The slow down is caused by a regression on scipy affecting versions 1.4.x: scipy/scipy#11299. The reason why only Windows runs are affected is because our CI config is pinning the version of scipy to 1.4.1 on Windows but not on Linux. This was introduced at #2328 (first commit of the file) and it's probably by accident that the two files are not aligned.

By unpinning the version and allowing scipy to use newer versions on Windows, the issue is fixed. This is an alternative solution to #3195.

@datumbox datumbox changed the base branch from master to tests/speedup December 19, 2020 18:44
@pytorch pytorch deleted a comment from codecov bot Dec 19, 2020
@codecov
Copy link

codecov bot commented Dec 19, 2020

Codecov Report

Merging #3196 (f3943d5) into tests/speedup (1a300d8) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@              Coverage Diff               @@
##           tests/speedup    #3196   +/-   ##
==============================================
  Coverage          73.48%   73.48%           
==============================================
  Files                 99       99           
  Lines               9230     9230           
  Branches            1476     1476           
==============================================
  Hits                6783     6783           
  Misses              1991     1991           
  Partials             456      456           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1a300d8...f3943d5. Read the comment docs.

@datumbox datumbox merged commit 7a028dd into pytorch:tests/speedup Dec 21, 2020
@datumbox datumbox mentioned this pull request Dec 21, 2020
@datumbox datumbox deleted the tests/speedup_win_inception_googlenet_scipy branch December 21, 2020 10:41
@datumbox
Copy link
Contributor Author

The failing tests are not related to this PR.

datumbox added a commit that referenced this pull request Jan 4, 2021
* Speedup test_ucf101 (#2623 

* Speedup Cmake builds (#3186)

* Speedup test_autoaugment (#3190)

* Speedup DeformConvTester (#3191)

* Speedup InceptionV3 and GoogleNet on Windows (#3196)
@fmassa
Copy link
Member

fmassa commented Jan 20, 2021

Awesome, thanks for the PR @datumbox !

facebook-github-bot pushed a commit that referenced this pull request Jan 21, 2021
Summary:
* Speedup test_ucf101 (#2623

* Speedup Cmake builds (#3186)

* Speedup test_autoaugment (#3190)

* Speedup DeformConvTester (#3191)

* Speedup InceptionV3 and GoogleNet on Windows (#3196)

Reviewed By: datumbox

Differential Revision: D25954568

fbshipit-source-id: bdcea84b112a9343f27619aef7036369598f631e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants