LibriTTS-460 training text And Training result

[WendongGan]

@yl4579 Hi, yl4579,I have a doubt. For LibriTTS-460 training text, you only gave examples from styletts1, and did not provide train_libritts_list.txt and val_libritts_list.txt of styletts2. But there is no speaker id in the example of styletts1. I want to confirm one question, for example: LibriTTS/train-clean-460/2113/132485/2113_132485_000061_000006.wav||aɪ wʌz kɹˈuːəl!

Did you train 2113_132485 as a speaker or 2113 as a speaker? At present, I use 2113 as a speaker, that is, I use the audio of different chapters of this speaker as a speaker. So far, I've been training from scratch, following your code, stage1 and stage2. config parameters, diff_epoch: 10, joint_epoch: 15, my stage2 has been trained to 25epoch, still worse than your pre-trained model (like 20epcoh, diff_epoch: 0, joint_epoch: 0), I'm wondering where I might be at odds with you.

Look forward your help. If possible, can you upload your train_libritts_list.txt and val_libritts_list.txt ?

[yl4579]

I have already uploaded the training data, which is in https://github.com/yl4579/StyleTTS2/blob/main/Data/OOD_texts.txt, including the speaker ids. I used the training texts from LibriTTS as the OOD texts as described in the paper.

Could you show me the training curve? The diff_epoch and joint_epoch setting to 0 was because I had to resume training so I just set them to a low number. They aren’t the actual epochs those training started.

[WendongGan]

@yl4579 hi, yl4579, Thanks for your reply!

This is my eval loss.

As for the train loss, it is a bit of a pity that my training was stopped by me at 25epoch yesterday. I continued the training from 25epcoh, but I looked in the code and the iters were set to 0, so the tensorboard I found was overwritten, only the most recent ones.

My train loss is :

[yl4579]

It doesn't look abnormal at all. What kind of audio did you get out of it?

[WendongGan]

In addition, I continued the training on your pre-trained model (epochs_2nd_00020.pth) and I still set diff_epoch: 0, joint_epoch: 0. Here is my training curve.

I found the eval mel_loss is only 0.37, but the one trained by myself is still 0.385.

[yl4579]

By the way, I found a bug in the code. The dur_loss and mel_loss is exactly the same for the evaluation loss.

I think looking at mel loss doesn't tell you a lot, but all your loss seems higher than what I got (including F0 loss). This says something might be off during training. What kind of speech sample did you get in the end?

[WendongGan]

The train loss curve looked pretty good before it was covered, and I show that this is covered, so only the recent iters. As for the effect, I think it is close to your pre-trained model, and some aspects are worse. For example: sometimes there are subtle noises, some of them are less expressive than yours. Just give me a second. I'll sort out and upload some comparison samples of my model and yours.

[WendongGan]

These is the sample generated by the model trained by mayself and your pretrained model.
https://drive.google.com/file/d/1vq4AWtX02tDuNsIpvvOCJqj2-5uJm-Ds/view?usp=sharing

I found that:
The audio from your pre-trained model is much more similar. My similarity is not stable.

[yl4579]

I have no access to the files.

[WendongGan]

sorry，I have modified it. It should be ready to download now

[WendongGan]

I have already uploaded the training data, which is in https://github.com/yl4579/StyleTTS2/blob/main/Data/OOD_texts.txt, including the speaker ids. I used the training texts from LibriTTS as the OOD texts as described in the paper.

Could you show me the training curve? The diff_epoch and joint_epoch setting to 0 was because I had to resume training so I just set them to a low number. They aren’t the actual epochs those training started.

I checked the text, My understanding and what i do is correct, that is, 2113 is regarded as a speaker, that is, the audio of different chapters from the same speaker is regarded as a speaker.

[yl4579]

@WendongGan I listened to the samples you sent me. I think the quality isn't that different, though it's indeed true that your model sounds less natural than mine in prosody. Where did you find these speakers? Are the reference speakers from LibriTTS or not? Are they seen speakers or not?

What's interesting is that the speaker 0011 speaks in British accent, and the pre-trained model could capture this yet your model couldn't. Did you use the latest code that has the st fixed? Honestly it sounds like the model trained with the previous code that contains a bug: #20 (comment)

[WendongGan]

@yl4579 The test sample is from ESD . it is not from LibriTTS.
Yes, I also think just the prosody of my sample is less natural than yours.

the Code of train_second.py I used is this one .

What I find different is here

The pre-training model you provided is really good, and the effect of my training is gradually approaching. What I want to verify is whether the current code can efficiently and quickly achieve the effect of your pre-trained model. When I do, my improvement attempts and other work may be more meaningful. I am trying the new code. Do you have any other advice or share? Thanks again!

[yl4579]

@WendongGan This is one of the mistakes but I don't think it'll make such a big difference. The model was produced using the code slightly different from the current version (mostly 32f760b this version with change here: 041ea6d). I don't think I have time to verify whether the current code can reproduce the pre-trained model quality, but I don't think the quality would change that much. Also the epoch number is not exactly 20 or anything because I reset the number and stopped it manually (it was only named as epoch 20). I think the best strategy is to listen to the samples periodically and stop at one that sounds the best because the SLM adversarial training could get unstable sometimes, but your problem seems deeper than that as the accent was not captured either, so it doesn't sound like a problem of SLM adversarial training. I would check the reconstruction in first stage to make sure it doesn't go wrong there.

Also, have you checked checkpoints in earlier epochs (like epochs right when the mel loss stops decreasing)? Maybe some earlier epochs should sound better?