Adding OverFlow

[shivammehta25]

This is the model from the paper: https://arxiv.org/abs/2211.06892
Audio samples: https://shivammehta25.github.io/OverFlow/

[CLAassistant]

All committers have signed the CLA.

[king-dahmanus]

Good idea, from the samples I've hird the thing is quite good. What about adding neural hmm, or is it the same thing but just upgraded?

[shivammehta25]

It shares neural hmm as its core instead of attention. The benefits of neural HMM TTS are that it's almost half the number of parameters and it works very well even in a low resource setting i.e when we don't have enough data to train on. Once we merge this it would require very little change to add the neural HMM TTS into the system, which I plan to do as well.

[king-dahmanus]

That's nice. How fast is it on the cpu? I previously suggested improving its speed for screen readers but I didn't realize how foolish that was untill recently. So how fast, or faster/slower is it compared to tacotron2 with hifigan or vits?

[erogol]

Cool the PR is ready. I'll first try the LJSpeech recipe and let you know how it goes.

[erogol]

If this is a blocker to use multi-gpu we should explain this in the model docstring and docs too

[erogol]

Is this a model specific issue or rooted from torch

[shivammehta25]

This is a torch issue gradient checkpointing in a loop is currently not supported in DDP. It works fine for Multi-GPU if we turn off the flag by use_grad_checkpointing=False but will significantly increase memory usage while training. This is because to compute the actual data likelihood (not an approximation using MAS/Viterbi) we must use all the states at the previous time step during the forward pass to decide the probability mass at the current step.

[shivammehta25]

I have added some information please take a look and see if it needs any more explanation.

[Edresson]

Great PR @shivammehta25 Thanks for the contribution :).

[shivammehta25]

Oh my bad! I clicked one too many time the request for review button. Sorry for the spam xP.
And @Edresson Thank you very much :D Glad you liked it!

[erogol]

@shivammehta25 how do you compute lj_parameters.pt?

[shivammehta25]

Inside TTS/tts/layers/overflow/comon_layers.py. There exists OverflowUtils.get_data_parameters_for_flat_start which is called in on_init_start of the model. It loads the training data and computes the means and std (a single scalar) and for the whole training set to standardise the data during training (the model has mean and std as registered_buffers). This is used during the initialization in OutputNet, where the weights of the last layer are set to zeros and bias is set to 0 and 1 (a cool hack from fixup initialization). It also computes an average transition probability to start with an "ideal diagonal alignment". I sort of "cache" these lj_parameters.pt them near the cached phonemes.

[erogol]

Ok thanks. I missed it for some reason. I also trained an LJSpeech model. It works great.

But I think we also need to train a vocoder that we can do separately.

Mergin it now 👍

[shivammehta25]

I tried synthesising waveforms with the universal hifigan vocoder it works pretty well!