[💡SUG] Add callback function hook for end of training an epoch

[tszumowski]

Is your feature request related to a problem? Please describe.
I do not currently see a built-in way to assign a callback function to run after an epoch completes. This is helpful for pytorch-native capabilities as well as custom uses. For example, I use callbacks on epoch-complete for external distributed hyperparameter optimization routines.

Describe the solution you'd like
Add a callback function hook into the trainers that provides epoch index and the validation score for use externally. See context section for an example patch. I'm happy to PR it in, or feel free to take as you see appropriate.

Describe alternatives you've considered
None considered.

Additional context

I attached a patch file that adds an optional callback_fn. Note: This patch only shows it for the Trainer class, not the child classes. That is because this is for an example only. Also, if adopting this, it may be worth adding some try/except protection around the callback function so that any malformed callback behavior doesn't impact the main training process. Since that is a design decision, I didn't include it in the example.

trainer.py.patch.txt

[chenyushuo]

Thanks for your insightful commment. We'd appreciate it if you would like to PR it in branch 0.1.x. By the way, any PRs are welcome.
Note: trainer.py has been refactor in #521 , be careful to add this feature.

[tszumowski]

@chenyushuo thank you for making this and the other PRs! I meant to submit them weeks ago, but I had to stop and move to a different project for a little while. Thank you again.