Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable stepwise processing flag for schedulers #806

Closed
peteriz opened this issue Feb 9, 2020 · 1 comment · Fixed by #941
Closed

Enable stepwise processing flag for schedulers #806

peteriz opened this issue Feb 9, 2020 · 1 comment · Fixed by #941
Labels
feature Is an improvement or enhancement help wanted Open to be worked on

Comments

@peteriz
Copy link

peteriz commented Feb 9, 2020

🚀 Feature

Asking if it makes sense adding a flag in the Trainer class for calling scheduler.step() after every update (per #640).

Motivation

This makes sense for training NLP models such as BERT/XLNet or any other that update the lr based on the current step (and training defined in terms of steps instead of epochs) instead of the current state that it is called after an epoch ends.
I'm aware that users can override optimizer_step of the model, however it's a quite common training pattern for training such NLP models.
I think this feature is worthwhile and I will contribute my changes. Let me know if not.

Pitch

Add scheduler.step() call after every step (modify the optimizer step accordingly)

@peteriz peteriz added feature Is an improvement or enhancement help wanted Open to be worked on labels Feb 9, 2020
@williamFalcon
Copy link
Contributor

instead of a flag, why not add the config to the scheduler? mostly because you might want to do different things to each scheduler.

maybe allow a dictionary of options?

Here's an option:

def configure_optimizers(...):
    return [adam], [{'scheduler': Scheduler(), "step_frequency": "epoch|batch"}]

@neggert @jeffling

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Is an improvement or enhancement help wanted Open to be worked on
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants