Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error while running BatchNorm on OSX CPU #624

Closed
sklan opened this issue Feb 15, 2019 · 5 comments
Closed

Error while running BatchNorm on OSX CPU #624

sklan opened this issue Feb 15, 2019 · 5 comments

Comments

@sklan
Copy link
Contributor

sklan commented Feb 15, 2019

The last test case fails on OSX CPU

BatchNorm: Test Failed at /Users/sklan/JuliaProjects/Flux.jl/test/layers/normalisation.jl:108
  Expression: #= /Users/sklan/JuliaProjects/Flux.jl/test/layers/normalisation.jl:108 =# @allocated(m(x)) < 100000000
   Evaluated: 3588496016 < 100000000
Stacktrace:
 [1] top-level scope at /Users/sklan/JuliaProjects/Flux.jl/test/layers/normalisation.jl:108
 [2] top-level scope at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083
 [3] top-level scope at /Users/sklan/JuliaProjects/Flux.jl/test/layers/normalisation.jl:33
Test Summary: | Pass  Fail  Total
BatchNorm     |   12     1     13
ERROR: LoadError: Some tests did not pass: 12 passed, 1 failed, 0 errored, 0 broken.
@MikeInnes
Copy link
Member

cc @KristofferC, ref #586. Looks like that performance fix isn't working there but I can't think why that would change.

@KristofferC
Copy link
Contributor

KristofferC commented Feb 15, 2019 •
edited
Loading

Pff... I have no idea either. @sklan could you report on what you get running

using Flux
bn = BatchNorm(32);
testimg = randn(Float32, 416, 416, 32, 1);
@time bn(testimg)
@time bn(testimg)

@sklan
Copy link
Contributor Author

sklan commented Feb 15, 2019

This is the output.

Last login: Fri Feb 15 21:26:25 on ttys003
MBP:~ sklan$ exec '/Applications/Julia-1.1.app/Contents/Resources/julia/bin/julia'
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.1.0 (2019-01-21)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Flux

julia> bn = BatchNorm(32);

julia> testimg = randn(Float32, 416, 416, 32, 1);

julia> @time bn(testimg)
 14.939357 seconds (225.12 M allocations: 3.799 GiB, 2.11% gc time)
Tracked 416×416×32×1 Array{Float32,4}:
[:, :, 1, 1] =
  1.0704      0.235531   -1.18566     …   1.3272     -1.13366   -0.0152019 
  0.30278     0.163673    1.14822        -0.525794   -0.108948   1.90977   
 -0.724658   -2.36177    -1.15598         0.124364   -0.314997  -0.713621  
 -1.7457      0.21635    -0.579613       -2.98781     1.19092   -0.255075  
  1.47016    -0.562483   -0.449468       -0.597041   -0.5848     1.26782   
 -1.03485    -2.13825    -0.784882    …  -0.484836   -2.93766    0.465792  
 -0.134129    0.433125   -1.79609         0.790999   -0.621625   1.12556   
 -0.0116278   1.30441    -0.315548       -1.00536    -0.16059   -0.233067  
 -1.62309    -0.199577    0.236489       -1.13757     1.37636   -0.00112643
 -1.61514    -0.808219   -0.0128811      -0.62011     1.06598    0.445637  
  ⋮                                   ⋱                          ⋮         
 -1.59548    -1.28994     0.0016553      -0.530933   -1.81411   -1.49638   
 -0.401066   -0.491277   -1.62854         0.269285   -0.324059  -0.0486714 
 -1.65511    -0.272479    0.184396       -0.332391    0.537881   0.283656  
  0.561506   -0.533602    0.549964    …   0.584174    1.23303    0.745588  
 -0.311751   -0.32383     1.57959        -0.0780646  -1.40084   -0.843616  
  0.493313    0.306215    0.2913          1.37252     0.765631  -0.147737  
 -0.965453   -0.0168756  -0.315536        0.167498   -0.188581  -0.132391  
  1.61106     0.0473542   0.00711687      0.844389    0.418887   0.216325  
 -0.487691   -1.05958    -0.754778    …  -0.749624   -1.28066    0.525504  

[:, :, 2, 1] =
  2.4354      -0.117401    0.725613  …   0.581743   1.49734   -0.0381739
  0.458637     0.959612    0.316605     -0.133536   0.264089   2.05913  
  0.0680593    0.0186288  -1.48838       0.932719   2.38814    1.3703   
  0.711957    -0.660863    0.896198      1.14144    1.03486    0.925498 
 -0.13253     -0.389323    1.03221      -1.10122   -1.49549   -1.38625  
  1.12263      0.109787   -2.40405   …  -0.942594   1.04436    2.61076  
  2.21194     -1.29929    -1.10496       1.32504   -1.45227    0.703579 
 -0.773897     0.571213   -0.813008     -0.904286   1.92065   -0.350453 
  1.36877      1.46371    -0.463008     -1.41889    1.19304    0.61148  
  0.654595    -1.62709    -1.90069       2.02916    0.241492   0.0400088
  ⋮                                  ⋱                         ⋮        
 -0.00282756  -0.513553   -0.171925     -0.828478   1.5977    -0.806207 
 -1.11601     -1.65426    -0.340676     -0.175749  -0.301899   1.22784  
  0.0696412   -0.54589    -1.03267       0.440066   1.15516   -0.174497 
 -1.63398     -2.00915    -1.2184    …   0.114133  -0.380731   0.190048 
 -0.528322    -0.712281    0.484895     -1.11701   -1.31083   -2.36139  
  0.162812     0.62855    -0.384679     -0.251337  -0.651474  -0.746967 
  0.458833     1.44868     0.756085     -0.59534    1.86526    0.506907 
 -0.98144     -0.464287   -1.45419      -0.275528  -1.06631    1.28555  
  1.51291      0.237346    0.245883  …  -1.22622   -0.440371  -2.57213  

[:, :, 3, 1] =
 -0.388959   -2.70969    0.531102   …   0.21368     -0.105238   -0.0894886
  0.181752   -1.50988    0.551657       0.00883099   0.0562058  -0.45321  
  0.235251    0.891571   0.0692665     -0.129787    -0.211329    0.666528 
 -0.94378     1.59579    1.72708       -1.44886     -2.33664     0.153273 
 -0.717832   -0.153899   0.737281      -1.82774      1.26092     1.56788  
  1.27574    -0.426442  -1.71373    …  -0.1561       1.48934    -2.5742   
  0.843281    1.86473   -1.05174       -0.750514    -1.00208     0.242921 
 -0.0535696  -0.911295   2.19918       -1.03847      1.30073    -2.19667  
  2.36884    -0.272986   0.684814      -0.972227    -0.812309    0.203837 
  0.803338    1.87141    0.219423      -1.25552     -0.048013    1.4469   
  ⋮                                 ⋱                            ⋮        
  1.73742     1.50053   -0.32758        0.509557     0.139862    0.286002 
 -2.09887    -0.449654   0.938805      -1.86986     -1.74977    -0.461885 
 -1.32907     0.296562  -0.775862       0.840906    -0.103144    0.321101 
  0.298828   -1.27614    0.58123    …  -0.573572    -0.270879   -1.20756  
  0.217878    2.03708   -1.43008       -0.0383055   -0.421442    0.793709 
  0.828814   -0.288958   1.01704        1.77124     -1.33972     0.458552 
  0.496226   -0.199797  -0.692842       0.584962     2.32083    -1.01457  
 -0.097998   -2.10214   -0.647122       0.254858     0.70077     1.26562  
 -0.263551    2.92696    0.870838   …   0.802764     1.32956     0.771013 

...

[:, :, 30, 1] =
 -0.798643    1.55108    -0.844603   …   0.2455    -0.749378    0.300164 
 -0.938348    1.62612     0.190818      -0.447224   1.40065     0.158022 
 -1.28977    -0.542148    1.24458       -1.08816   -0.424959    1.64279  
  0.199855   -0.0874816   0.585716       1.24783    0.51564    -0.79076  
 -0.815182   -0.256682   -0.769924       0.967085   0.607035    0.946782 
  1.00862     0.503713    0.0938944  …  -0.294353  -0.285737    0.0220266
 -1.96074     0.684936   -0.388022       0.996808  -0.618669    0.165742 
 -0.649457    1.03012    -1.00918        1.65662   -0.847128    1.7973   
 -0.0478125   0.640823    0.254411       0.572402   2.08985     0.681957 
 -0.257465   -0.599677    0.218351      -0.656554   1.01091     0.72693  
  ⋮                                  ⋱                          ⋮        
  0.647815   -0.0397092  -1.25101        2.01431    1.35636    -0.648056 
 -0.930146   -0.494276    1.15318        1.28857   -0.726446    0.880347 
  2.53377     1.5723      1.05229        0.147631  -0.353543   -0.539161 
  0.0993184  -1.03929    -0.308662   …  -0.959379  -0.54964    -0.875858 
 -1.04715    -0.387905   -0.881656       0.961712  -0.402917    0.58097  
  1.09332    -0.191712   -0.779859      -0.701822   0.47084     0.654515 
  0.142847   -0.559475    1.28886        0.652706   0.691471   -1.50701  
  0.746706    1.15737    -1.75737        1.14106   -1.15155     1.53317  
 -0.834962    0.760783    0.565803   …   0.36484    0.0419386   0.908328 

[:, :, 31, 1] =
  0.508998  -0.924782  -0.496361   …  -0.177071    -1.46082     0.717032 
 -0.157874  -1.78888   -0.711145       0.375858     0.216875   -0.36463  
  1.6102     0.117904  -1.00793        0.0738172   -1.07027     0.209384 
  0.466743  -1.7206    -0.361045      -0.747969     0.596635    0.267792 
 -0.97376    0.832288   0.961763      -0.987079     2.02408     0.147116 
 -0.282242   0.61551   -0.142156   …  -0.255006    -0.377585    0.747731 
  1.97398   -0.298081  -0.0305993      0.139028     0.215057    1.40875  
 -0.782039  -1.61527    0.760976       0.568258     0.979689   -0.713478 
  1.81861    0.376789   0.405441       0.304708     0.951165   -0.392045 
  0.231625   0.645468  -1.58355        1.85518      0.139929   -1.18528  
  ⋮                                ⋱                            ⋮        
  0.863103  -0.677762  -1.14733        0.00751001   1.07698     1.50509  
  0.32262    0.554421  -3.62124       -0.177157     0.233521   -0.0955126
  1.55227   -0.441285   1.96508       -0.652199     0.810965   -0.813488 
  1.00666    0.185664  -0.400837   …  -2.2943       1.47464     1.09072  
  0.429074   0.285961  -0.310323       0.748661     0.0453702   1.24927  
 -0.609856   1.16589   -0.224855      -0.585734     1.23793     1.2361   
  0.778301   0.249011   0.946499       0.723588     0.408373   -0.980806 
  1.89904    0.684239   0.62053        0.844803    -1.09306     0.149259 
 -0.22009   -2.66513   -0.131603   …   0.228216     0.30786     0.309769 

[:, :, 32, 1] =
  0.924057   -0.425161  -2.89858    …  -0.451558   -0.341647  -1.40744  
  1.37684     0.116819  -0.258081      -1.17043     0.152617   0.430696 
 -0.486293   -0.318874  -1.34248       -0.706969   -0.864055  -0.332836 
 -0.426044    1.31842    1.31501       -1.26336    -0.012672   0.233976 
 -0.408328    1.57448    2.2818         1.88857     2.18778    0.156917 
 -1.03048    -1.98418   -2.00387    …  -1.25135     2.54995    1.17266  
  0.188485    0.372426   0.71583       -1.1119     -1.10292    0.0411803
  2.21532    -0.530246   0.387045      -0.684772   -1.04815    0.35156  
  0.700847    0.045614  -0.690362       0.592488   -0.702275   1.15602  
 -0.379843   -0.468761   0.256661      -0.0375624  -0.521739  -0.667699 
  ⋮                                 ⋱                          ⋮        
  1.97647    -0.23817   -0.0324779      1.03638     0.157483   2.18409  
 -0.664579   -1.48024   -1.96688       -0.295725   -0.613148   0.0195059
  0.0407231   0.338034  -0.446905       2.47103    -0.323986  -0.0114705
  1.02231    -0.544984   0.256024   …  -1.06617     0.59007    1.01152  
  0.425429    0.6138    -1.3218        -0.0696797   0.584007  -0.724312 
 -0.588474   -0.437547   0.0276476      1.63645     0.610616   0.179348 
 -1.31685    -1.74603    1.18834       -0.75128     0.212967  -0.991186 
  0.700412    1.08395    0.288382       1.42702     0.570615  -0.285736 
  2.8269     -0.754285   1.54661    …   0.372398   -0.212199  -1.19372  

julia> @time bn(testimg)
 12.857950 seconds (215.97 M allocations: 3.342 GiB, 1.32% gc time)
Tracked 416×416×32×1 Array{Float32,4}:
[:, :, 1, 1] =
  1.0704      0.235531   -1.18566     …   1.3272     -1.13366   -0.0152019 
  0.30278     0.163673    1.14822        -0.525794   -0.108948   1.90977   
 -0.724658   -2.36177    -1.15598         0.124364   -0.314997  -0.713621  
 -1.7457      0.21635    -0.579613       -2.98781     1.19092   -0.255075  
  1.47016    -0.562483   -0.449468       -0.597041   -0.5848     1.26782   
 -1.03485    -2.13825    -0.784882    …  -0.484836   -2.93766    0.465792  
 -0.134129    0.433125   -1.79609         0.790999   -0.621625   1.12556   
 -0.0116278   1.30441    -0.315548       -1.00536    -0.16059   -0.233067  
 -1.62309    -0.199577    0.236489       -1.13757     1.37636   -0.00112643
 -1.61514    -0.808219   -0.0128811      -0.62011     1.06598    0.445637  
  ⋮                                   ⋱                          ⋮         
 -1.59548    -1.28994     0.0016553      -0.530933   -1.81411   -1.49638   
 -0.401066   -0.491277   -1.62854         0.269285   -0.324059  -0.0486714 
 -1.65511    -0.272479    0.184396       -0.332391    0.537881   0.283656  
  0.561506   -0.533602    0.549964    …   0.584174    1.23303    0.745588  
 -0.311751   -0.32383     1.57959        -0.0780646  -1.40084   -0.843616  
  0.493313    0.306215    0.2913          1.37252     0.765631  -0.147737  
 -0.965453   -0.0168756  -0.315536        0.167498   -0.188581  -0.132391  
  1.61106     0.0473542   0.00711687      0.844389    0.418887   0.216325  
 -0.487691   -1.05958    -0.754778    …  -0.749624   -1.28066    0.525504  

[:, :, 2, 1] =
  2.4354      -0.117401    0.725613  …   0.581743   1.49734   -0.0381739
  0.458637     0.959612    0.316605     -0.133536   0.264089   2.05913  
  0.0680593    0.0186288  -1.48838       0.932719   2.38814    1.3703   
  0.711957    -0.660863    0.896198      1.14144    1.03486    0.925498 
 -0.13253     -0.389323    1.03221      -1.10122   -1.49549   -1.38625  
  1.12263      0.109787   -2.40405   …  -0.942594   1.04436    2.61076  
  2.21194     -1.29929    -1.10496       1.32504   -1.45227    0.703579 
 -0.773897     0.571213   -0.813008     -0.904286   1.92065   -0.350453 
  1.36877      1.46371    -0.463008     -1.41889    1.19304    0.61148  
  0.654595    -1.62709    -1.90069       2.02916    0.241492   0.0400088
  ⋮                                  ⋱                         ⋮        
 -0.00282756  -0.513553   -0.171925     -0.828478   1.5977    -0.806207 
 -1.11601     -1.65426    -0.340676     -0.175749  -0.301899   1.22784  
  0.0696412   -0.54589    -1.03267       0.440066   1.15516   -0.174497 
 -1.63398     -2.00915    -1.2184    …   0.114133  -0.380731   0.190048 
 -0.528322    -0.712281    0.484895     -1.11701   -1.31083   -2.36139  
  0.162812     0.62855    -0.384679     -0.251337  -0.651474  -0.746967 
  0.458833     1.44868     0.756085     -0.59534    1.86526    0.506907 
 -0.98144     -0.464287   -1.45419      -0.275528  -1.06631    1.28555  
  1.51291      0.237346    0.245883  …  -1.22622   -0.440371  -2.57213  

[:, :, 3, 1] =
 -0.388959   -2.70969    0.531102   …   0.21368     -0.105238   -0.0894886
  0.181752   -1.50988    0.551657       0.00883099   0.0562058  -0.45321  
  0.235251    0.891571   0.0692665     -0.129787    -0.211329    0.666528 
 -0.94378     1.59579    1.72708       -1.44886     -2.33664     0.153273 
 -0.717832   -0.153899   0.737281      -1.82774      1.26092     1.56788  
  1.27574    -0.426442  -1.71373    …  -0.1561       1.48934    -2.5742   
  0.843281    1.86473   -1.05174       -0.750514    -1.00208     0.242921 
 -0.0535696  -0.911295   2.19918       -1.03847      1.30073    -2.19667  
  2.36884    -0.272986   0.684814      -0.972227    -0.812309    0.203837 
  0.803338    1.87141    0.219423      -1.25552     -0.048013    1.4469   
  ⋮                                 ⋱                            ⋮        
  1.73742     1.50053   -0.32758        0.509557     0.139862    0.286002 
 -2.09887    -0.449654   0.938805      -1.86986     -1.74977    -0.461885 
 -1.32907     0.296562  -0.775862       0.840906    -0.103144    0.321101 
  0.298828   -1.27614    0.58123    …  -0.573572    -0.270879   -1.20756  
  0.217878    2.03708   -1.43008       -0.0383055   -0.421442    0.793709 
  0.828814   -0.288958   1.01704        1.77124     -1.33972     0.458552 
  0.496226   -0.199797  -0.692842       0.584962     2.32083    -1.01457  
 -0.097998   -2.10214   -0.647122       0.254858     0.70077     1.26562  
 -0.263551    2.92696    0.870838   …   0.802764     1.32956     0.771013 

...

[:, :, 30, 1] =
 -0.798643    1.55108    -0.844603   …   0.2455    -0.749378    0.300164 
 -0.938348    1.62612     0.190818      -0.447224   1.40065     0.158022 
 -1.28977    -0.542148    1.24458       -1.08816   -0.424959    1.64279  
  0.199855   -0.0874816   0.585716       1.24783    0.51564    -0.79076  
 -0.815182   -0.256682   -0.769924       0.967085   0.607035    0.946782 
  1.00862     0.503713    0.0938944  …  -0.294353  -0.285737    0.0220266
 -1.96074     0.684936   -0.388022       0.996808  -0.618669    0.165742 
 -0.649457    1.03012    -1.00918        1.65662   -0.847128    1.7973   
 -0.0478125   0.640823    0.254411       0.572402   2.08985     0.681957 
 -0.257465   -0.599677    0.218351      -0.656554   1.01091     0.72693  
  ⋮                                  ⋱                          ⋮        
  0.647815   -0.0397092  -1.25101        2.01431    1.35636    -0.648056 
 -0.930146   -0.494276    1.15318        1.28857   -0.726446    0.880347 
  2.53377     1.5723      1.05229        0.147631  -0.353543   -0.539161 
  0.0993184  -1.03929    -0.308662   …  -0.959379  -0.54964    -0.875858 
 -1.04715    -0.387905   -0.881656       0.961712  -0.402917    0.58097  
  1.09332    -0.191712   -0.779859      -0.701822   0.47084     0.654515 
  0.142847   -0.559475    1.28886        0.652706   0.691471   -1.50701  
  0.746706    1.15737    -1.75737        1.14106   -1.15155     1.53317  
 -0.834962    0.760783    0.565803   …   0.36484    0.0419386   0.908328 

[:, :, 31, 1] =
  0.508998  -0.924782  -0.496361   …  -0.177071    -1.46082     0.717032 
 -0.157874  -1.78888   -0.711145       0.375858     0.216875   -0.36463  
  1.6102     0.117904  -1.00793        0.0738172   -1.07027     0.209384 
  0.466743  -1.7206    -0.361045      -0.747969     0.596635    0.267792 
 -0.97376    0.832288   0.961763      -0.987079     2.02408     0.147116 
 -0.282242   0.61551   -0.142156   …  -0.255006    -0.377585    0.747731 
  1.97398   -0.298081  -0.0305993      0.139028     0.215057    1.40875  
 -0.782039  -1.61527    0.760976       0.568258     0.979689   -0.713478 
  1.81861    0.376789   0.405441       0.304708     0.951165   -0.392045 
  0.231625   0.645468  -1.58355        1.85518      0.139929   -1.18528  
  ⋮                                ⋱                            ⋮        
  0.863103  -0.677762  -1.14733        0.00751001   1.07698     1.50509  
  0.32262    0.554421  -3.62124       -0.177157     0.233521   -0.0955126
  1.55227   -0.441285   1.96508       -0.652199     0.810965   -0.813488 
  1.00666    0.185664  -0.400837   …  -2.2943       1.47464     1.09072  
  0.429074   0.285961  -0.310323       0.748661     0.0453702   1.24927  
 -0.609856   1.16589   -0.224855      -0.585734     1.23793     1.2361   
  0.778301   0.249011   0.946499       0.723588     0.408373   -0.980806 
  1.89904    0.684239   0.62053        0.844803    -1.09306     0.149259 
 -0.22009   -2.66513   -0.131603   …   0.228216     0.30786     0.309769 

[:, :, 32, 1] =
  0.924057   -0.425161  -2.89858    …  -0.451558   -0.341647  -1.40744  
  1.37684     0.116819  -0.258081      -1.17043     0.152617   0.430696 
 -0.486293   -0.318874  -1.34248       -0.706969   -0.864055  -0.332836 
 -0.426044    1.31842    1.31501       -1.26336    -0.012672   0.233976 
 -0.408328    1.57448    2.2818         1.88857     2.18778    0.156917 
 -1.03048    -1.98418   -2.00387    …  -1.25135     2.54995    1.17266  
  0.188485    0.372426   0.71583       -1.1119     -1.10292    0.0411803
  2.21532    -0.530246   0.387045      -0.684772   -1.04815    0.35156  
  0.700847    0.045614  -0.690362       0.592488   -0.702275   1.15602  
 -0.379843   -0.468761   0.256661      -0.0375624  -0.521739  -0.667699 
  ⋮                                 ⋱                          ⋮        
  1.97647    -0.23817   -0.0324779      1.03638     0.157483   2.18409  
 -0.664579   -1.48024   -1.96688       -0.295725   -0.613148   0.0195059
  0.0407231   0.338034  -0.446905       2.47103    -0.323986  -0.0114705
  1.02231    -0.544984   0.256024   …  -1.06617     0.59007    1.01152  
  0.425429    0.6138    -1.3218        -0.0696797   0.584007  -0.724312 
 -0.588474   -0.437547   0.0276476      1.63645     0.610616   0.179348 
 -1.31685    -1.74603    1.18834       -0.75128     0.212967  -0.991186 
  0.700412    1.08395    0.288382       1.42702     0.570615  -0.285736 
  2.8269     -0.754285   1.54661    …   0.372398   -0.212199  -1.19372  

julia>

@KristofferC
Copy link
Contributor

No need to paste the whole resulting array :) Seems like it is very slow though.

@KristofferC
Copy link
Contributor

Locally I get:

> @time bn(testimg);
  1.840182 seconds (9.07 M allocations: 524.891 MiB, 8.59% gc time)

> @time bn(testimg);
  0.038541 seconds (178 allocations: 63.382 MiB, 27.92% gc time)

@DhairyaLGandhi DhairyaLGandhi mentioned this issue Feb 20, 2019
Merged
@sklan sklan closed this as completed Mar 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@KristofferC @MikeInnes @sklan