Earlystopping callback metrics in different devices with single gpu

[Les1ie]

The device of the metric return by validation_step is GPU, related code is

def validation_step(self, batch, batch_idx):
    x, y = batch
    if y.device != self.device:
        y = y.to(self.device)
    y_hat = self(x)
    loss = self.loss(y_hat, y)    # loss.device is cuda.
    self.log('valid loss', loss.item())
    return loss

After an epoch of validation compeleted when using earlystopping, follow error occured:

  File "E:\Python\Python37\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 871, in run_train
    self.train_loop.run_training_epoch()
  File "E:\Python\Python37\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 584, in run_training_epoch
    self.trainer.run_evaluation(on_epoch=True)
  File "E:\Python\Python37\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1011, in run_evaluation
    self.evaluation_loop.on_evaluation_end()
  File "E:\Python\Python37\lib\site-packages\pytorch_lightning\trainer\evaluation_loop.py", line 102, in on_evaluation_end
    self.trainer.call_hook('on_validation_end', *args, **kwargs)
  File "E:\Python\Python37\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1228, in call_hook
    trainer_hook(*args, **kwargs)
  File "E:\Python\Python37\lib\site-packages\pytorch_lightning\trainer\callback_hook.py", line 227, in on_validation_end
    callback.on_validation_end(self, self.lightning_module)
  File "E:\Python\Python37\lib\site-packages\pytorch_lightning\callbacks\early_stopping.py", line 173, in on_validation_end
    self._run_early_stopping_check(trainer)
  File "E:\Python\Python37\lib\site-packages\pytorch_lightning\callbacks\early_stopping.py", line 193, in _run_early_stopping_check
    should_stop, reason = self._evalute_stopping_criteria(current, trainer)
  File "E:\Python\Python37\lib\site-packages\pytorch_lightning\callbacks\early_stopping.py", line 226, in _evalute_stopping_criteria
    elif self.monitor_op(current - self.min_delta, self.best_score.to(trainer.lightning_module.device)):
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

This error didn't appear until I updated the version of pytorch_lightning.

[awaelchli]

Looking into it in #8295

[Les1ie]

Thanks