resume_from_checkpoint should not start from scratch if ckpt is not found

[devashishd12]

🐛 Bug

If the checkpoint file is not found at the location provided in resume_from_checkpoint argument in pl.Trainer, the training starts from scratch after displaying a UserWarning that is easy to miss.

To Reproduce

Use the following BoringModel.

Expected behavior

Should raise a FileNotFoundError and not start training from scratch.

[awaelchli]

Not sure if there is a real reason why we have warning instead of error. Want to give this a try? Contributions are welcome.

[ia-davidpichler]

I actually preferred the old semantics of this parameter as it makes my logic for an interruptible training job easier. Also the docs weren't updated to reflect this change: https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html#init

[carmocca]

Also the docs weren't updated to reflect this change: https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html#init

@ia-davidpichler Would you be interested in opening a PR with the doc fix? 😄

[russellbrooks]

Agreed that the prior behavior is better for accommodating interruptible training jobs (e.g. AWS Spot) and this feels like a regression in capability.

Whether to error or warn should be exposed as an additional parameter, with it defaulting to something like error_on_checkpoint_missing=False for consistency with docs:

If there is no checkpoint file at the path, start from scratch. If resuming from mid-epoch checkpoint, training will start from the beginning of the next epoch.

[carmocca]

The discussion for the change is in the PR that closed this: #7075

I'll update the docs.