How can I check which epoch is picked for the best model?

[ssrokyz]

During a training process, it keeps updating the best_model.pth file.
So how can I be sure, which epoch is picked for the best model?

[Linux-cpp-lisp]

At the end of each epoch, the trainer checks whether the current validation metrics are better than those of the last best_model.pth. If they are, it overwrites best_model.pth and prints

! Best model     [epochnum]    [metrics_value]

to the log.

You can find out which epoch best_model.pth corresponds to by checking for the last instance of this line in the log file, something like:

cat root/run_name/log | grep "\! Best model" | tail -n 1

giving, for example,

! Best model     1872    0.001

which means best_model.pth contains the model as of epoch 1872.

[ssrokyz]

Oh, How did I miss it. 😂 Thank you very much @Linux-cpp-lisp !

[DQM520]

At the end of each epoch, the trainer checks whether the current validation metrics are better than those of the last best_model.pth. If they are, it overwrites best_model.pth and prints

! Best model     [epochnum]    [metrics_value]

to the log.

You can find out which epoch best_model.pth corresponds to by checking for the last instance of this line in the log file, something like:

cat root/run_name/log | grep "\! Best model" | tail -n 1

giving, for example,

! Best model     1872    0.001

which means best_model.pth contains the model as of epoch 1872.

I see from the full.yaml example the default epoch is 100000(max_epochs: 100000 # stop training after _ number of epochs, we set a very large number here, it won't take this long in practice and we will use early stopping instead), however, my dadatset is very large,which has 578731 structures, and time consumption in one epoch is about 4 hours. I don't know how long will all the epochs doned. All I do is wait until the early stopping based on metrics values triggers? Or I can stop the epoch manually?