Streaming state-full inference example

[KDuzinkiewicz]

Hi! I'm new to tract but I stumbled upon it when looking for a way to turn my simple PyTorch U-Net model into something that can run efficiently on a CPU. I have exported my model to ONNX format and was able to run a piece of Rust code to run inference using tract. It works but I want to make it more efficient. AFAIU, the way to go, is to use streaming inference approach. My model takes a noisy magnitude spectrogram as input and transforms it into a de-noised one. The problem is that when I operate in real-time I get a single frame of that spectrogram every 10ms. Since the receptive field of my model is 16 frames I could simply keep a 16-element circular buffer and run the inference on it every time a new frame comes in, but this is not optimal - many calculations in the network are redundant. AFAIU, tract offers a way to handle the internal states of convolutional layers, etc. and enable the user to pass a single frame for inference. The problem is I don't really know how to code this - I can't find any example illustrating this approach :( Can someone point me in the right direction? Thanks in advance for any help.