Improving BatchNorm

[sklan]

Fixes 2 issues

1. ϵ is declared but never used.
2. Makes BatchNorm notation similar to that of the original paper with similar performance.

Ubuntu 18.04 GPU K-80

Test

bn = gpu(BatchNorm(32))
testimg = gpu(randn(Float32, 416, 416, 32, 1))
@time gpu(bn(testimg))
@time gpu(bn(testimg))

Old Code

3.327136 seconds (3.25 M allocations: 167.726 MiB, 1.83% gc time)
0.001245 seconds (155 allocations: 5.391 KiB)

New Code

3.542283 seconds (3.46 M allocations: 178.192 MiB, 2.74% gc time)
0.001185 seconds (149 allocations: 5.281 KiB)

OSX CPU

Test

using Flux
bn = BatchNorm(32)
testimg = randn(Float32, 416, 416, 32, 1)
@time bn(testimg)
@time bn(testimg)

Old Code

14.939357 seconds (225.12 M allocations: 3.799 GiB, 2.11% gc time)
12.857950 seconds (215.97 M allocations: 3.342 GiB, 1.32% gc time)

New Code

2.701686 seconds (8.37 M allocations: 487.830 MiB, 9.61% gc time)
0.077704 seconds (135 allocations: 63.381 MiB, 24.40% gc time)

[DhairyaLGandhi]

Ref: #624
Thanks for the PR :)

[KristofferC]

Locally I get

julia> @time bn(testimg);
  0.032135 seconds (170 allocations: 63.381 MiB, 15.82% gc time)

for old code on cpu. Not sure why you still have the superslow version even after #586. All CI also passes the allocation test so something seems odd with your installation. Anyway, if this fixes it might as well go with this.

[staticfloat]

This seems fine to me, but I'm not sure how it would have such a performance difference. @sklan what version are you testing when you say "old"?

[sklan]

@staticfloat I am using Pkg.dev that clones the master branch.
The performance difference is only on the mac cpu and I can't figure out why this works either.

[MikeInnes]

GPU tests pass locally, so I think we can take this. Thanks a lot @sklan!