Not-yet-existing resume_from_checkpoint for auto-resubmit

[tarepan]

🚀 Feature

Accept Not-yet-existing resume_from_checkpoint in Trainer for automatic training resume / auto-resubmit.

Motivation

In cloud ML training services (e.g. Google AI platform training, AWS SageMaker, AWS Batch), there are Job auto-retry feature.
If we can specify checkpoint path, Job auto-retry can be used for training resume / resubmit.
Unfortunately, PyTorch-Lightning cannot specify Non-(yet-)existing file as resume_from_checkpoint argument of Trainer, it simply raise an error.
The motivation of this feature request is enabling training resume through Not-yet-existing resume_from_checkpoint.
(This feature looks similar to auto-resubmit of pl's SLURM. but I am totally newbie about it, it could be nonsense.)

Pitch

current checkpoint restore process:
https://github.com/PyTorchLightning/pytorch-lightning/blob/3abfec896212ea85e45d6ac3ccb323ef242d16de/pytorch_lightning/trainer/connectors/checkpoint_connector.py#L57-L60
It use (existing) resume_from_checkpoint.
If specified path's file is empty, it raise an error.

What I hope is

         1. HPC weights. 
         2. if no HPC weights ""try to"" restore checkpoint_path weights  
         3. otherwise don't restore weights 

It means that if checkpoint_path (resume_from_checkpoint) file do not exist, simply ignore it and start training from scratch.
In this case, training start normally from scratch, then pl save checkpoints.
If we set save-path == resume_from_checkpoint, latest checkpoint file exist in resume_from_checkpoint path.
When job auto-retry is triggered, because now checkpoint file exists in resume_from_checkpoint, in retried job pl load checkpoint from resume_from_checkpoint, so training properly resume.

Alternatives

Use hpc_save & hpc_load's resume system for normal training.
As far as I read the codes, "HPC weights load" (for slurm...?) enable auto-resubmit based on directory (not file) + file name rule (hpc_ckpt_{ckpt_number}.ckpt).
If we accept checkpoint directory (e.g. resume_from_checkpoint_dir), same mechanism can be used for resume/resubmit.

[tarepan]

Add draft pull request for discussion.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[tarepan]

wait for merge.

[tarepan]

The PR is merged.
Thanks for all efforts of contributers.