update checkpoint docs

[Borda]

What does this PR do?

Fixes #775. Revisit checkpoint documentation...

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[pep8speaks]

Hello @Borda! Thanks for updating this PR.

• In the file pytorch_lightning/callbacks/model_checkpoint.py:

Line 24:101: E501 line too long (122 > 100 characters)
Line 25:1: W293 blank line contains whitespace
Line 98:101: E501 line too long (105 > 100 characters)
Line 173:101: E501 line too long (110 > 100 characters)
Line 199:101: E501 line too long (102 > 100 characters)

• In the file tests/test_restore_models.py:

Line 136:101: E501 line too long (104 > 100 characters)
Line 348:101: E501 line too long (104 > 100 characters)

Comment last updated at 2020-03-03 20:15:02 UTC

[jeremyjordan]

looks like there are some lingering references tocheckpoint_callback.filepath in:

• tests/test_restore_models.py
• tests/models/utils.py
• tests/trainer/test_trainer.py

[williamFalcon]

@Borda mind implementing the rich text for file names?

[Borda]

shall we make it rather model-name_epoch=02_val_loss=0.36.ckpt and if there is already something with model-name then it creates model-name-v1_epoch=02_val_loss=0.36.ckpt that would make sense more... but we can move it to next release and just keep this one now @williamFalcon

[williamFalcon]

________________________________________________________________________________ test_resume_from_checkpoint_epoch_restored ________________________________________________________________________________

tmpdir = local('/tmp/pytest-of-waf251/pytest-48/test_resume_from_checkpoint_ep0')

    def test_resume_from_checkpoint_epoch_restored(tmpdir):
        """Verify resuming from checkpoint runs the right number of epochs"""
        import types

        tutils.reset_seed()

        hparams = tutils.get_hparams()

        def _new_model():
            # Create a model that tracks epochs and batches seen
            model = LightningTestModel(hparams)
            model.num_epochs_seen = 0
            model.num_batches_seen = 0

            def increment_epoch(self):
                self.num_epochs_seen += 1

            def increment_batch(self, _):
                self.num_batches_seen += 1

            # Bind the increment_epoch function on_epoch_end so that the
            # model keeps track of the number of epochs it has seen.
            model.on_epoch_end = types.MethodType(increment_epoch, model)
            model.on_batch_start = types.MethodType(increment_batch, model)
            return model

        model = _new_model()

        trainer_options = dict(
            show_progress_bar=False,
            max_epochs=2,
            train_percent_check=0.65,
            val_percent_check=1,
            checkpoint_callback=ModelCheckpoint(tmpdir, save_top_k=-1),
            logger=False,
            default_save_path=tmpdir,
            early_stop_callback=False,
            val_check_interval=0.5,
        )

        # fit model
        trainer = Trainer(**trainer_options)
        trainer.fit(model)

        training_batches = trainer.num_training_batches

        assert model.num_epochs_seen == 2
        assert model.num_batches_seen == training_batches * 2

        # Other checkpoints can be uncommented if/when resuming mid-epoch is supported
        checkpoints = sorted(glob.glob(os.path.join(trainer.checkpoint_callback.dirpath, '*.ckpt')))

        for check in checkpoints[::2]:
            next_model = _new_model()
            state = torch.load(check)

            # Resume training
            trainer_options['max_epochs'] = 4
            new_trainer = Trainer(**trainer_options, resume_from_checkpoint=check)
            new_trainer.fit(next_model)
>           assert state['global_step'] + next_model.num_batches_seen == training_batches * 4
E           assert 140 == 160
E             -140
E             +160

[Borda]

it is hard to replicate it locally, it seems the names on server are different...

[Borda]

it shall be fixed now, @williamFalcon ^^