Persistent RNNs

[oleg-trott]

Baidu found a way to get decent throughput in RNNs at very low batch sizes (like 4).

It would be interesting to see that in Neon/NervanaGPU

[jennifermyers]

Hi Oleg,

Thanks for the note! We have added support for this internally and it is currently in test. We expect to release it in an upcoming version of neon - possibly not the imminent 1.5 release, but likely by 1.6.

[oleg-trott]

That was fast. I thought that paper just came out.

[thestew42]

Greg gave a talk at GTC in April on persistent RNN kernels which we have been working to implement.

[jennifermyers]

Good news! This is now expected to be part of the imminent v1.5.0 release.