Allocate workspace repeatedly right before calling into CUDNN.

[maleadt]

This to deal with GC runs affecting CUDNN workspace size heuristics.

cc @MikeInnes

[MikeInnes]

One other possibility is to have a CuArrays equivalent of GC.enable to just disallow memory recovery while doing these allocations.

One concern with using anonymous functions like this is that it could end up producing a lot of untyped slots, but hopefully it's simple enough to avoid that.

[maleadt]

One other possibility is to have a CuArrays equivalent of GC.enable to just disallow memory recovery while doing these allocations.

It's only going to actually free stuff when it's really needed though (to prevent OOM), so we can't disable that. And disabling the Julia GC won't do anything, because those finalizers only put stuff back in the pool, and don't actually free (for now).

One concern with using anonymous functions like this is that it could end up producing a lot of untyped slots, but hopefully it's simple enough to avoid that.

The code looks good, fully typed and all closures inlined. Pretty bloaty still, but not significantly worse with these extra closures.

[appleparan]

Code looks okay but it seems just CI system failure. please rerun?

[appleparan]

Suddenly, I feel dangerous to have possibility of infinite loop. How about change while to for loop and break in @workspace?

[MikeInnes]

It could only be infinite if the required workspace increases forever. Not vouching for CUDNN but at any rate, the memory allocation will error out eventually.

[maleadt]

Did some more work on @workspace, it has some additional features now. @MikeInnes, your fear was right and use of closures leads to badly inferred code due to the capturing issue. In this case, I think we can safely deconstruct the closure into a let block though.

[maleadt]

This should be good to go if CI passes. Might also be the time for some feedback on the @workspace and @argout macros (cc @kshyatt, you've wrapped a lot of these), but those improvements can land in another PR since this isn't a public API.