Call any trainer function from the LightningCLI

CHANGELOG.md

@@ -10,6 +10,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 ### Added
 
 
+- Added support to call any trainer function from the `LightningCLI` ([#7508](https://github.com/PyTorchLightning/pytorch-lightning/pull/7508))
+
+
 - Added `clip_grad_by_value` support for TPUs ([#7025](https://github.com/PyTorchLightning/pytorch-lightning/pull/7025))
 
 
@@ -35,13 +38,16 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Changed the behaviour when logging evaluation step metrics to no longer append `/epoch_*` to the metric name ([#7351](https://github.com/PyTorchLightning/pytorch-lightning/pull/7351))
 
 
-- Changed `resolve_training_type_plugins` to allow setting `num_nodes` and `sync_batchnorm` from `Trainer` setting ([7026](https://github.com/PyTorchLightning/pytorch-lightning/pull/7026))
+- Changed `resolve_training_type_plugins` to allow setting `num_nodes` and `sync_batchnorm` from `Trainer` setting ([#7026](https://github.com/PyTorchLightning/pytorch-lightning/pull/7026))
+
+
+- Default `seed_everything(workers=True)` in the `LightningCLI` ([#7504](https://github.com/PyTorchLightning/pytorch-lightning/pull/7504))
 
 
-- Changed `model.state_dict()` in `CheckpointConnector` to allow `training_type_plugin` to customize the model's `state_dict()` ([7474](https://github.com/PyTorchLightning/pytorch-lightning/pull/7474))
+- Changed `model.state_dict()` in `CheckpointConnector` to allow `training_type_plugin` to customize the model's `state_dict()` ([#7474](https://github.com/PyTorchLightning/pytorch-lightning/pull/7474))
 
 
-- MLflowLogger now uses the env variable `MLFLOW_TRACKING_URI` as default tracking uri ([7457](https://github.com/PyTorchLightning/pytorch-lightning/pull/7457))
+- MLflowLogger now uses the env variable `MLFLOW_TRACKING_URI` as default tracking uri ([#7457](https://github.com/PyTorchLightning/pytorch-lightning/pull/7457))
 
 
 ### Deprecated

docs/source/common/lightning_cli.rst

@@ -7,8 +7,8 @@
     from pytorch_lightning.core.datamodule import LightningDataModule
     from pytorch_lightning.utilities.cli import LightningCLI
 
-    original_fit = LightningCLI.fit
-    LightningCLI.fit = lambda self: None
+    original_run = LightningCLI.run
+    LightningCLI.run = lambda self: None
 
     class MyModel(LightningModule):
         def __init__(
@@ -37,7 +37,7 @@
 
 .. testcleanup:: *
 
-    LightningCLI.fit = original_fit
+    LightningCLI.run = original_run
     mock_argv.stop()
 
 
@@ -91,8 +91,22 @@ practice to create a configuration file and provide this to the tool. A way to d
 
 The instantiation of the :class:`~pytorch_lightning.utilities.cli.LightningCLI` class takes care of parsing command line
 and config file options, instantiating the classes, setting up a callback to save the config in the log directory and
-finally running :func:`trainer.fit`. The resulting object :code:`cli` can be used for instance to get the result of fit,
-i.e., :code:`cli.fit_result`.
+finally running the trainer. The resulting object :code:`cli` can be used for instance to get the instance of the
+model, i.e., :code:`cli.model`.
+
+The :class:`~pytorch_lightning.utilities.cli.LightningCLI` is configured to run
+:meth:`~pytorch_lightning.trainer.Trainer.fit` by default. This can be changed either by setting
+``LightningCLI(trainer_fn="test")`` or by passing the argument through command line positionally, e.g.:
+
+.. code-block:: bash
+
+    python trainer.py test --trainer.limit_test_batches=10
+
+.. tip::
+
+    You can override :meth:`~pytorch_lightning.utilities.cli.LightningCLI.prepare_run_kwargs` to pass any extra
+    arguments to the trainer function to run.
+
 
 After multiple trainings with different configurations, each run will have in its respective log directory a
 :code:`config.yaml` file. This file can be used for reference to know in detail all the settings that were used for each
@@ -341,9 +355,9 @@ The :class:`~pytorch_lightning.utilities.cli.LightningCLI` class has the
 :meth:`~pytorch_lightning.utilities.cli.LightningCLI.add_arguments_to_parser` method which can be implemented to include
 more arguments. After parsing, the configuration is stored in the :code:`config` attribute of the class instance. The
 :class:`~pytorch_lightning.utilities.cli.LightningCLI` class also has two methods that can be used to run code before
-and after :code:`trainer.fit` is executed: :meth:`~pytorch_lightning.utilities.cli.LightningCLI.before_fit` and
-:meth:`~pytorch_lightning.utilities.cli.LightningCLI.after_fit`. A realistic example for these would be to send an email
-before and after the execution of fit. The code would be something like:
+and after the trainer runs: :meth:`~pytorch_lightning.utilities.cli.LightningCLI.before_run` and
+:meth:`~pytorch_lightning.utilities.cli.LightningCLI.after_run`. A realistic example for these would be to send an email
+before and after the execution. The code would be something like:
 
 .. testcode::
 
@@ -354,23 +368,24 @@ before and after the execution of fit. The code would be something like:
         def add_arguments_to_parser(self, parser):
             parser.add_argument('--notification_email', default='will@email.com')
 
-        def before_fit(self):
+        def before_run(self):
             send_email(
                 address=self.config['notification_email'],
-                message='trainer.fit starting'
+                message='Trainer running'
             )
 
-        def after_fit(self):
+        def after_run(self):
             send_email(
                 address=self.config['notification_email'],
-                message='trainer.fit finished'
+                message='Trainer finished'
             )
 
     cli = MyLightningCLI(MyModel)
 
 Note that the config object :code:`self.config` is a dictionary whose keys are global options or groups of options. It
 has the same structure as the yaml format described previously. This means for instance that the parameters used for
-instantiating the trainer class can be found in :code:`self.config['trainer']`.
+instantiating the trainer class can be found in :code:`self.config['trainer']`. You can also access the trainer function
+that is meant to run with :code:`self.config["trainer_fn"].value`.
 
 Another case in which it might be desired to extend :class:`~pytorch_lightning.utilities.cli.LightningCLI` is that the
 model and data module depend on a common parameter. For example in some cases both classes require to know the

pl_examples/basic_examples/autoencoder.py

@@ -116,8 +116,7 @@ def test_dataloader(self):
 
 def cli_main():
     cli = LightningCLI(LitAutoEncoder, MyDataModule, seed_everything_default=1234)
-    result = cli.trainer.test(cli.model, datamodule=cli.datamodule)
-    print(result)
+    cli.trainer.test(cli.model, datamodule=cli.datamodule)
 
 
 if __name__ == '__main__':

pl_examples/basic_examples/backbone_image_classifier.py

@@ -129,8 +129,7 @@ def test_dataloader(self):
 
 def cli_main():
     cli = LightningCLI(LitClassifier, MyDataModule, seed_everything_default=1234)
-    result = cli.trainer.test(cli.model, datamodule=cli.datamodule)
-    print(result)
+    cli.trainer.test(cli.model, datamodule=cli.datamodule)
 
 
 if __name__ == '__main__':

pl_examples/basic_examples/dali_image_classifier.py

@@ -222,8 +222,7 @@ def cli_main():
         return
 
     cli = LightningCLI(LitClassifier, MyDataModule, seed_everything_default=1234)
-    result = cli.trainer.test(cli.model, datamodule=cli.datamodule)
-    print(result)
+    cli.trainer.test(cli.model, datamodule=cli.datamodule)
 
 
 if __name__ == "__main__":

pl_examples/basic_examples/simple_image_classifier.py

@@ -77,8 +77,7 @@ def configure_optimizers(self):
 
 def cli_main():
     cli = LightningCLI(LitClassifier, MNISTDataModule, seed_everything_default=1234)
-    result = cli.trainer.test(cli.model, datamodule=cli.datamodule)
-    print(result)
+    cli.trainer.test(cli.model, datamodule=cli.datamodule)
 
 
 if __name__ == '__main__':

pytorch_lightning/accelerators/accelerator.py

@@ -369,7 +369,7 @@ def setup_optimizers(self, trainer: 'pl.Trainer') -> None:
         Args:
             trainer: the Trainer, these optimizers should be connected to
         """
-        if trainer.state.fn not in (TrainerFn.FITTING, TrainerFn.TUNING):
+        if trainer.state.fn not in (TrainerFn.fit, TrainerFn.tune):
             return
         optimizers, lr_schedulers, optimizer_frequencies = self.training_type_plugin.init_optimizers(
             trainer=trainer, model=self.lightning_module

pytorch_lightning/callbacks/early_stopping.py

@@ -159,7 +159,7 @@ def on_load_checkpoint(self, callback_state: Dict[str, Any]) -> None:
 
     def _should_skip_check(self, trainer) -> bool:
         from pytorch_lightning.trainer.states import TrainerFn
-        return trainer.state.fn != TrainerFn.FITTING or trainer.sanity_checking
+        return trainer.state.fn != TrainerFn.fit or trainer.sanity_checking
 
     def on_train_epoch_end(self, trainer, pl_module) -> None:
         if not self._check_on_train_epoch_end or self._should_skip_check(trainer):

pytorch_lightning/callbacks/model_checkpoint.py

@@ -305,7 +305,7 @@ def _should_skip_saving_checkpoint(self, trainer: 'pl.Trainer') -> bool:
         from pytorch_lightning.trainer.states import TrainerFn
         return (
             trainer.fast_dev_run  # disable checkpointing with fast_dev_run
-            or trainer.state.fn != TrainerFn.FITTING  # don't save anything during non-fit
+            or trainer.state.fn != TrainerFn.fit  # don't save anything during non-fit
             or trainer.sanity_checking  # don't save anything during sanity check
             or self._last_global_step_saved == trainer.global_step  # already saved at the last step
         )

pytorch_lightning/plugins/training_type/ddp_spawn.py

@@ -280,7 +280,7 @@ def transfer_distrib_spawn_state_on_fit_end(self, results):
             # save the last weights
             last_path = None
             if (
-                self.lightning_module.trainer.state.fn == TrainerFn.FITTING and best_model_path is not None
+                self.lightning_module.trainer.state.fn == TrainerFn.fit and best_model_path is not None
                 and len(best_model_path) > 0
             ):
                 last_path = re.sub(".ckpt", ".tmp_end.ckpt", best_model_path)
@@ -298,7 +298,7 @@ def __recover_child_process_weights(self, best_path, last_path):
         # todo, pass also best score
 
         # load last weights
-        if last_path is not None and self.lightning_module.trainer.state.fn == TrainerFn.FITTING:
+        if last_path is not None and self.lightning_module.trainer.state.fn == TrainerFn.fit:
             ckpt = pl_load(last_path, map_location=lambda storage, loc: storage)
             self.lightning_module.load_state_dict(ckpt)
 

pytorch_lightning/plugins/training_type/deepspeed.py

@@ -532,7 +532,7 @@ def restore_model_state_from_ckpt_path(
         if not self.save_full_weights and self.world_size > 1:
             # Rely on deepspeed to load the checkpoint and necessary information
             from pytorch_lightning.trainer.states import TrainerFn
-            is_fitting = self.lightning_module.trainer.state.fn == TrainerFn.FITTING
+            is_fitting = self.lightning_module.trainer.state.fn == TrainerFn.fit
             save_dir = self._filepath_to_dir(ckpt_path)
 
             if self.zero_stage_3:

pytorch_lightning/plugins/training_type/rpc_sequential.py

@@ -208,7 +208,7 @@ def _skip_init_connections(self):
         Returns: Whether to skip initialization
 
         """
-        return torch_distrib.is_initialized() and self.lightning_module.trainer.state.fn != TrainerFn.FITTING
+        return torch_distrib.is_initialized() and self.lightning_module.trainer.state.fn != TrainerFn.fit
 
     def init_model_parallel_groups(self):
         num_model_parallel = 1  # TODO currently no support for vertical model parallel
@@ -231,7 +231,7 @@ def _infer_check_num_gpus(self):
         return self.world_size
 
     def handle_transferred_pipe_module(self) -> None:
-        if self.lightning_module.trainer.state.fn == TrainerFn.FITTING:
+        if self.lightning_module.trainer.state.fn == TrainerFn.fit:
             torch_distrib.barrier()  # Ensure we await main process initialization
             # Add trainer/configure_optimizers to the pipe model for access in all worker processes
             rpc_pipe.PipeModel.trainer = self.lightning_module.trainer
@@ -243,7 +243,7 @@ def init_pipe_module(self) -> None:
         # Create pipe_module
         model = self.lightning_module
         self._find_and_init_pipe_module(model)
-        if self.lightning_module.trainer.state.fn == TrainerFn.FITTING:
+        if self.lightning_module.trainer.state.fn == TrainerFn.fit:
             torch_distrib.barrier()  # Ensure we join main process initialization
             model.sequential_module.foreach_worker(register_optimizers, include_self=True)
 

pytorch_lightning/plugins/training_type/sharded.py

@@ -66,7 +66,7 @@ def _reinit_optimizers_with_oss(self):
         trainer.convert_to_lightning_optimizers()
 
     def _wrap_optimizers(self):
-        if self.model.trainer.state.fn != TrainerFn.FITTING:
+        if self.model.trainer.state.fn != TrainerFn.fit:
             return
         self._reinit_optimizers_with_oss()
 

pytorch_lightning/plugins/training_type/sharded_spawn.py

@@ -53,7 +53,7 @@ def _reinit_optimizers_with_oss(self):
         trainer.optimizers = optimizers
 
     def _wrap_optimizers(self):
-        if self.model.trainer.state.fn != TrainerFn.FITTING:
+        if self.model.trainer.state.fn != TrainerFn.fit:
             return
         self._reinit_optimizers_with_oss()
 

pytorch_lightning/plugins/training_type/tpu_spawn.py

@@ -190,7 +190,7 @@ def transfer_distrib_spawn_state_on_fit_end(self, results):
             # save the last weights
             last_path = None
             if (
-                self.lightning_module.trainer.state.fn == TrainerFn.FITTING and best_model_path is not None
+                self.lightning_module.trainer.state.fn == TrainerFn.fit and best_model_path is not None
                 and len(best_model_path) > 0
             ):
                 last_path = re.sub(".ckpt", ".tmp_end.ckpt", best_model_path)

pytorch_lightning/trainer/configuration_validator.py

@@ -31,14 +31,14 @@ def verify_loop_configurations(self, model: 'pl.LightningModule') -> None:
             model: The model to check the configuration.
 
         """
-        if self.trainer.state.fn in (TrainerFn.FITTING, TrainerFn.TUNING):
+        if self.trainer.state.fn in (TrainerFn.fit, TrainerFn.tune):
             self.__verify_train_loop_configuration(model)
             self.__verify_eval_loop_configuration(model, 'val')
-        elif self.trainer.state.fn == TrainerFn.VALIDATING:
+        elif self.trainer.state.fn == TrainerFn.validate:
             self.__verify_eval_loop_configuration(model, 'val')
-        elif self.trainer.state.fn == TrainerFn.TESTING:
+        elif self.trainer.state.fn == TrainerFn.test:
             self.__verify_eval_loop_configuration(model, 'test')
-        elif self.trainer.state.fn == TrainerFn.PREDICTING:
+        elif self.trainer.state.fn == TrainerFn.predict:
             self.__verify_predict_loop_configuration(model)
         self.__verify_dp_batch_transfer_support(model)
 

pytorch_lightning/trainer/connectors/logger_connector/epoch_result_store.py

@@ -355,7 +355,7 @@ def update_logger_connector(self) -> Tuple[Dict, Dict]:
 
         # TODO(carmocca): when we implement flushing the logger connector metrics after
         # the trainer.state changes, this should check trainer.evaluating instead
-        if self.trainer.state.fn in (TrainerFn.TESTING, TrainerFn.VALIDATING):
+        if self.trainer.state.fn in (TrainerFn.test, TrainerFn.validate):
             logger_connector.evaluation_callback_metrics.update(callback_metrics)
 
         # update callback_metrics

pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py

@@ -282,7 +282,7 @@ def get_evaluate_epoch_results(self) -> _EVALUATE_OUTPUT:
 
         # log results of evaluation
         if (
-            self.trainer.state.fn != TrainerFn.FITTING and self.trainer.evaluating and self.trainer.is_global_zero
+            self.trainer.state.fn != TrainerFn.fit and self.trainer.evaluating and self.trainer.is_global_zero
             and self.trainer.verbose_evaluate
         ):
             print('-' * 80)

pytorch_lightning/trainer/evaluation_loop.py

@@ -101,7 +101,7 @@ def on_evaluation_end(self, *args: Any, **kwargs: Any) -> None:
         else:
             self.trainer.call_hook('on_validation_end', *args, **kwargs)
 
-        if self.trainer.state.fn != TrainerFn.FITTING:
+        if self.trainer.state.fn != TrainerFn.fit:
             # summarize profile results
             self.trainer.profiler.describe()
 

pytorch_lightning/trainer/states.py

@@ -35,20 +35,20 @@ class TrainerFn(LightningEnum):
     such as :meth:`~pytorch_lightning.trainer.trainer.Trainer.fit` and
     :meth:`~pytorch_lightning.trainer.trainer.Trainer.test`.
     """
-    FITTING = 'fit'
-    VALIDATING = 'validate'
-    TESTING = 'test'
-    PREDICTING = 'predict'
-    TUNING = 'tune'
+    fit = 'fit'
+    validate = 'validate'
+    test = 'test'
+    predict = 'predict'
+    tune = 'tune'
 
     @property
     def _setup_fn(self) -> 'TrainerFn':
         """
-        ``FITTING`` is used instead of ``TUNING`` as there are no "tune" dataloaders.
+        ``fit`` is used instead of ``tune`` as there are no "tune" dataloaders.
 
         This is used for the ``setup()`` and ``teardown()`` hooks
         """
-        return TrainerFn.FITTING if self == TrainerFn.TUNING else self
+        return TrainerFn.fit if self == TrainerFn.tune else self
 
 
 class RunningStage(LightningEnum):
@@ -58,11 +58,11 @@ class RunningStage(LightningEnum):
     This stage complements :class:`TrainerFn` by specifying the current running stage for each function.
     More than one running stage value can be set while a :class:`TrainerFn` is running:
 
-        - ``TrainerFn.FITTING`` - ``RunningStage.{SANITY_CHECKING,TRAINING,VALIDATING}``
-        - ``TrainerFn.VALIDATING`` - ``RunningStage.VALIDATING``
-        - ``TrainerFn.TESTING`` - ``RunningStage.TESTING``
-        - ``TrainerFn.PREDICTING`` - ``RunningStage.PREDICTING``
-        - ``TrainerFn.TUNING`` - ``RunningStage.{TUNING,SANITY_CHECKING,TRAINING,VALIDATING}``
+        - ``TrainerFn.fit`` - ``RunningStage.{SANITY_CHECKING,TRAINING,VALIDATING}``
+        - ``TrainerFn.validate`` - ``RunningStage.VALIDATING``
+        - ``TrainerFn.test`` - ``RunningStage.TESTING``
+        - ``TrainerFn.predict`` - ``RunningStage.PREDICTING``
+        - ``TrainerFn.tune`` - ``RunningStage.{TUNING,SANITY_CHECKING,TRAINING,VALIDATING}``
     """
     TRAINING = 'train'
     SANITY_CHECKING = 'sanity_check'