Usage for TTS task

[lexkoro]

I was thinking about using model2vec to provide semantic information to the TextEncoder in a TTS task. Wondering if this would be a valid approach and which model would be best suited for a multilingual TTS.

[Pringled]

Hi @lexkoro,

I'm not too familiar with TTS personally, but I think the task is most likely too difficult for a static embedder like ours. We do have a multilingual model available here which you could experiment with though, curious to hear about your results!

[lexkoro]

Hey,

currently I am conditioning the LayerNorm (RMSNorm) in the TextEncoder with the embeddings coming from a distilled BAAI/bge-m3. The tts model itself sounds fine, but I am not sure how much impact (good or bad) the embeddings actually have and if they provide relevant semantic sentence information to the model.

PS: Thanks for the project btw! :)

[Pringled]

Sounds like an interesting usecase! It's not something we have experimented with ourselves, but the multilingual model I linked might work well for you, at least to initialize the embedding layer.