Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tensorboard example outputs. #337

Closed
erogol opened this issue Jan 14, 2020 · 8 comments
Closed

Tensorboard example outputs. #337

erogol opened this issue Jan 14, 2020 · 8 comments

Comments

@erogol
Copy link
Contributor

erogol commented Jan 14, 2020

Below is how a healty training looks like. I put it here as a reference for people using TTS. This is a training on LJSpeech with default model configuration.
scalars_tb
healty_images_tb
audios_tb

@IveJ
Copy link

IveJ commented Jan 14, 2020

How about your input data

@erogol
Copy link
Contributor Author

erogol commented Jan 15, 2020

@IveJ what do you like to know ?

@IveJ
Copy link

IveJ commented Jan 15, 2020 via email

@erogol
Copy link
Contributor Author

erogol commented Jan 15, 2020

All of these possible but I can't really comment without knowing all your data and training. Please post it on discourse.

@vcjob
Copy link

vcjob commented Jan 15, 2020

Hi @erogol
according to the plots, what's the right step to stop training? Is it still learning at 300k+ step? what's the main plot we are looking at to decide when to stop? I mean, it seems like on the stop_loss plot it does not learn any longer, the training is going on though. Wouldn't it hurt the final model performance?

@erogol
Copy link
Contributor Author

erogol commented Jan 15, 2020

@vcjob https://github.com/mozilla/TTS/wiki/FAQ#how-do-i-know-when-to-stop-training

@WeberJulian
Copy link
Contributor

WeberJulian commented May 12, 2020

Hi, I have a similar looking tensorboard but sometime on the ground truth, I have a weird artefact that is maybe just the silence trim but the models is struggling to replicate the green rectangle. Is this normal behaviour ? Shouldn't it be just deep blue if it's silence ? Can it hurt performance ?

Screenshot 2020-05-12 at 11 43 53

EDIT : If anyone has the same question @erogol responded here

@heoun
Copy link

heoun commented Sep 22, 2020

@erogol - I'm running into an issue with tensorboard where "No scalar data was found." even though the output is the same as the one in the config.json. Any idea on how to resolve this issue? I have already ran the model and wanted to confirm the performance of my model via tensorboard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants