Buildkite CI failures with grad test of ConvTranspose + selu

[ToucheSir]

One example: https://buildkite.com/julialang/flux-dot-jl/builds/1914#c62d9761-ab7f-415a-b995-51552eb2b1e5

I could not repro this locally on 2 separate GPU machines, master and 2 threads (same number Buildkite uses). Oddly, it's succeded twice (once on trying and once on staging), but fails with the same result every time, which makes me suspect some environmental discrepancy between workers.

[ToucheSir]

Despite pulling the exact inputs and weights from failing buildkite runs, I was not able to replicate this locally. If someone has a znver2 machine they can test on, everything needed should be in https://gist.github.com/ToucheSir/32fd6688d3932c9f498c78a42a0ea017. @DhairyaLGandhi and possibly @maleadt, is there any way to replicate the CI environment and run tests against that? I don't think it's necessary to have a GPU attached, because the inconsistencies are coming from the CPU calculation. I read the test output backwards, the discrepancy is in the GPU results.

[ToucheSir]

I could get the same summed result by exporting the pre-selu output and loading that locally, but still have not been able to generate a similar output and wasn't able to coax a container image into downloading the same dep versions. What's confusing is that the tests for ConvTranspose + identity activation do pass on CI, yet the differing output for the selu version is happening before the activation function is applied. Either way, #1822 is now up as a stopgap measure until some brave soul decides to revisit this in the future.

[ToucheSir]

Fixed by #1836, ref. #1836 (comment).