incorrect run on the test set with overwritten validation_end and test_epoch_end

[philip-bl]

🐛 Bug

If I override validation_end and test_epoch_end, TrainerEvaluationLoopMixin.evaluate works incorrectly on the test set

Suppose we override validation_epoch_end and test_end, but not validation_end and test_epoch_end. (I actually did this since I am a newbie and haven't yet figured out how everything works; also it seems validation_end is the same as validation_epoch_end, and test_end seems to be the same as test_epoch_end). Suppose I run trainer.test(model). Consider lines 300-312 in evaluation_loop.py. Then we have (test_mode and self.is_overriden('test_end', model=model)) == True, so the first if block is executed, that is eval_results = model.test_end(outputs). But look at the second if and its elif. We have (test_mode and self.is_overriden('test_epoch_end', model=model)) == False, hence the elif of the second if will also be executed, that is eval_results = model.validation_epoch_end(outputs). And we will have validation results recorder as test results, which is a mistake.

This problem is present in the commit 60b8246. And the inverse problem (which happens if we override only test_epoch_end and validation_end is present in 0.7.1.

[github-actions]

Hi! thanks for your contribution!, great first issue!

[awaelchli]

I can look into this.
To summarize, there is a problem when we mix *_epoch_end (new version) and *_end (old naming) for validation and test. There is no problem if we would consequently use only one of the two, either the deprecated or the new way. Is this correct?

I propose this fix:

if test_mode:
    if is_overriden("test_end"):
        # get results and warn user
    elif is_overriden("test_epoch_end"):
        # get results
else: 
    # same as above but with "validation_end", "validation_epoch_end"

[philip-bl]

@awaelchli yes