Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace a MisconfigurationException with warning in ModelCheckpoint callback #4560

Merged
merged 6 commits into from
Nov 10, 2020

Conversation

kazhang
Copy link
Contributor

@kazhang kazhang commented Nov 6, 2020

What does this PR do?

I have a custom checkpoint callback that inherits from the ModelCheckpoint callback to support every_n_steps by saving all(save_top_k=-1 and monitor=None) and skip if not every n steps ((global_step + 1) % every_n_steps != 0)

At the same time, I also want to store the last checkpoint(save_last=True) so that we could resume training from crash.

However, this exception prevent me from doing so.

This PR replace the exception with a warning as discussed in #4335

Before submitting

  • Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?
  • Did you verify new and existing tests pass locally with your changes?
  • If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In in short, see following bullet-list:

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified; Bugfixes should be including in bug-fix release milestones (m.f.X) and features should be included in (m.X.b) releases.

Did you have fun?

Make sure you had fun coding 🙃

@codecov
Copy link

codecov bot commented Nov 7, 2020

Codecov Report

Merging #4560 (713e37f) into master (465ec75) will decrease coverage by 0%.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #4560   +/-   ##
======================================
- Coverage      93%     93%   -0%     
======================================
  Files         116     116           
  Lines        8855    8854    -1     
======================================
- Hits         8248    8244    -4     
- Misses        607     610    +3     

@Borda Borda added the refactor label Nov 8, 2020
Comment on lines 297 to 300
with pytest.raises(
MisconfigurationException, match=r'ModelCheckpoint\(save_last=True, monitor=None\) is not a valid.*'
):
ModelCheckpoint(dirpath=tmpdir, save_last=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can convert this to pytest.warns(UserWarning, ...

Copy link
Contributor

@tchaton tchaton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch !

Copy link
Member

@SkafteNicki SkafteNicki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@SkafteNicki SkafteNicki merged commit 30ad3e2 into Lightning-AI:master Nov 10, 2020
@kazhang kazhang deleted the relax-checkpoint-check branch November 10, 2020 18:13
@edenlightning edenlightning added this to the 1.0.x milestone Nov 10, 2020
SeanNaren pushed a commit that referenced this pull request Nov 11, 2020
…allback (#4560)

* replace MisconfigurationException with warning

* update test

* check raising UserWarning

Co-authored-by: chaton <thomas@grid.ai>
Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>
(cherry picked from commit 30ad3e2)
Borda pushed a commit that referenced this pull request Nov 11, 2020
…allback (#4560)

* replace MisconfigurationException with warning

* update test

* check raising UserWarning

Co-authored-by: chaton <thomas@grid.ai>
Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>
(cherry picked from commit 30ad3e2)
rohitgr7 pushed a commit that referenced this pull request Nov 21, 2020
…allback (#4560)

* replace MisconfigurationException with warning

* update test

* check raising UserWarning

Co-authored-by: chaton <thomas@grid.ai>
Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants