Delta neural networks inference

[casper2002casper]

Describe the potential feature

In scenarios where only part of the input layer is augmented, delta neural networks can be used to speed up inference by only updating the nodes affected by the change. The implementation of this principle has for example revolutionized chess engines (https://www.chessprogramming.org/NNUE), and has recently been implemented for CNNs https://github.com/facebookresearch/DeltaCNN.

Motivation

Could offer speedups in situations where the input layer is only partially updated, such as chess engines or combinatorial search methods.
Recent progress in CUDA sparse matrixes makes this interesting to evaluate.

Possible Implementation

Store hidden layer data from previous evaluation
Calculate input layer delta with previous evaluation and store in spare matrix
Update hidden layer data via sparse layer propagation

https://github.com/facebookresearch/DeltaCNN

[casper2002casper]

Might be better as a seperate module