Finetune in Spanish and Synthesize in Spanish and English

[ara-vardanyan]

Hi. I have a use case that requires synthesizing in multiple languages (Spanish and English in this case) in the same sentence or chunk of text. For example: "Yo puedo hablar en Espanol and I can speak in English as well".

Dataset:

• 20 hours of Spanish audio data (Mix between Argentinian, Colombian, Venezuelan, Puerto Rico speakers)
• I can get 20 hours of Spanish audio in just one dialect if needed

Approach

• Use Multilingual PL-BERT

Questions:

• Is fine-tuning instead of training from scratch the correct approach? (I assume so)
• Should I include English data in the fine-tuning in order for it to be able to speak both languages or will it still be able to speak English because of the pre-training? If so, what is a good ratio?
• When I run inference, how should I navigate around specifying the language? Since I need multiple languages, would I have to detect from the text what language each chunk is and phonemize them separately?

Greatly appreciate any help with this! Also if anyone already has an open source model of this sort, please let me know!

[rodrigoGA]

Were you able to make any progress? I have a similar case