You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Asking if it makes sense adding a flag in the Trainer class for calling scheduler.step() after every update (per #640).
Motivation
This makes sense for training NLP models such as BERT/XLNet or any other that update the lr based on the current step (and training defined in terms of steps instead of epochs) instead of the current state that it is called after an epoch ends.
I'm aware that users can override optimizer_step of the model, however it's a quite common training pattern for training such NLP models.
I think this feature is worthwhile and I will contribute my changes. Let me know if not.
Pitch
Add scheduler.step() call after every step (modify the optimizer step accordingly)
The text was updated successfully, but these errors were encountered:
🚀 Feature
Asking if it makes sense adding a flag in the Trainer class for calling
scheduler.step()
after every update (per #640).Motivation
This makes sense for training NLP models such as BERT/XLNet or any other that update the lr based on the current step (and training defined in terms of steps instead of epochs) instead of the current state that it is called after an epoch ends.
I'm aware that users can override
optimizer_step
of the model, however it's a quite common training pattern for training such NLP models.I think this feature is worthwhile and I will contribute my changes. Let me know if not.
Pitch
Add
scheduler.step()
call after every step (modify the optimizer step accordingly)The text was updated successfully, but these errors were encountered: