Can this be used in real time with microphone?

[mattedminster]

I am wondering if this can be used in real time with a microphone?

[snakers4]

Hi,

Technically speaking there is nothing preventing you from trying that. The model should work fine with decent audio from microphone.

But there are multiple facets to this:

• The model is just feed forward, and for proper operation / batching it is better to apply some VAD before the model (i.e. see TorchAudio vad);
• In terms of compute, current models should be fast enough to operate much faster than realtime on modern CPUs with proper feeding techniques, we will publish some benchmarks on commodity hardware;
• In terms of quality - there are similar domains in the train dataset;

[mattedminster]

Will give this a shot! thank you!

[snakers4]

As a first approximation this, together with VAD was added to colab example,
We are working on making tf-js work as well.

[snakers4]

Some further examples with microphone - https://github.com/snakers4/silero-vad/tree/master/examples

[meir-abergel]

I tested Silero V3 in a GStreamer/ONNX live audio transcription module.
The CPU consumption with the ONNX parameters out of the box is about 3Gflops/second audio on CPU (an i7-10700 can transcribe 120 audio wav streams in parallel in real time at full capacity). Tensorflow might give a performance boost.
This is without using VAD and a sample size of 64k, with buffer overlap for incomplete words

Nice job!

[snakers4]

The CPU consumption with the ONNX parameters out of the box is about 3Gflops/second audio on CPU (an i7-10700 can transcribe 120 audio wav streams in parallel in real time at full capacity).

Please note that you can use the quantized models and try model freezing from PyTorch 1.9, that may give a further 2-3x speed boost - https://t.me/snakers4/2763.
In ONNX quantized model export was not available when models were exported last time.

Tensorflow might give a performance boost

Given the huge improvements in PyTorch for production, I see very little reason to invest time in the TF ecosystem at all.
Each time I touched it in the past, I was repelled by its nature.

With PyTorch we were able to secure the following gains, all other things kept equal:

• Quantization - x2 speed boost
• Model fusion - 25% boost
• Model freezing - 15% boost

[meir-abergel]

Sorry, I meant TF over ONNX.
Thx for the quantization numbers! Trying

[AshutoshDongare]

Check out my repo that uses selaro models and rast to create a conversationalist https://github.com/AshutoshDongare/convo