Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model training #12

Open
rose-jinyang opened this issue Oct 15, 2020 · 3 comments
Open

Model training #12

rose-jinyang opened this issue Oct 15, 2020 · 3 comments

Comments

@rose-jinyang
Copy link

Hello
How are you?
Thanks for contributing this project.
I am training a new model with config model_mobilenetv2_with_two_auxiliary_losses and AISegment dataset for 8 days.
But it is strange that the best model is not updated for 7 days.
What do u think the reason?

@shivSD
Copy link

shivSD commented Nov 16, 2020

@rose-jinyang did you figure out the issue. Even we are having the same issue, Loss is not decreasing after awhile.

@dpcq
Copy link

dpcq commented Mar 12, 2021

i also have this problem ,change the lr and train many epoch (1000 at least)

@DAVID-Hown
Copy link

DAVID-Hown commented Aug 19, 2021

Hello, could you please tell me that I made a mistake when retraining the model? Do you have any good solutions?
load model init finish...
===========> training <===========
Epoch: [0][0/23] Lr-deconv: [0.0] Lr-other: [0.001] Loss 13.0424 (13.0424)
Traceback (most recent call last):
File "train.py", line 809, in
main(args)
File "train.py", line 774, in main
train(dataLoader_train, netmodel, optimizer, epoch, logger_train, exp_args)
File "train.py", line 519, in train
logger.scalar_summary(tag, value, step=i)
File "/data/data_hao/PFLD-pytorch/20210817/PortraitNet/util/logger.py", line 26, in scalar_summary
tf.summary.scalar(tag, value, step=step)
File "/root/.local/lib/python3.6/site-packages/tensorboard/plugins/scalar/summary_v2.py", line 49, in scalar
with tf.summary.summary_scope(
AttributeError: module 'tensorboard.summary._tf.summary' has no attribute 'summary_scope'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants