Standardized Testing for All Optimizers

[hyang0129]

References #992.

Aim is to create a single test file that tests all relevant classes within the optimizer module of tfa. The test will discover classes that inherit from tf.keras.optimizers.Optimizer then execute a set of standard tests on each of those classes.

This will test for common behaviors that optimizers should have (eg. serialization, minimizes, etc.).

[hyang0129]

@seanpmorgan the optimizer module contains many classes that do not behave in the same way as an optimizer object. For example, multi_optimizer is a wrapper instead of an optimizer.

Perhaps we could have a class signature that exempts certain classes from the standard test?

Anyways, I've written some code to discover classes and it should work for any module. I think this can extend to layers and other modules that all contain classes that inherit from a common parent.

[hyang0129]

Maybe we should have a list of behaviors that each class in a module should have. So for optimizers, I would imagine that list includes minimizes and serializes. Ironically, I feel like the process of defining tests for optimizers is defining what a tfa optimizer should be.

[hyang0129]

on failure we get something like this.

standard_test.py FF [100%]

================================== FAILURES ===================================
____________________ test_optimizer_minimize[cpu-LazyAdam] ____________________

[seanpmorgan]

Really like this and they way we can extend it to other subpackages as you mentioned. Couple of thoughts below.

[hyang0129]

@seanpmorgan I've updated the code based on your feedback.

I've defined two standard behaviors for opts which is minimizes and serializes. I feel like there should be more standard behaviors but idk.

Also, half the classes appear to be wrappers. Once wrappers are standardized, we may need a way to select only optimizers and exclude wrappers because the testing criteria for wrappers could be different (or not because aside from wrapping, the wrapper should have the minimize and serialize behaviors. maybe just change how the test instances the wrappers?).

[hyang0129]

Also, if we want to apply this to other modules, I think we should move the discover classes function to something more common like test_utils

[hyang0129]

@seanpmorgan

please review

[seanpmorgan]

LGTM thanks @hyang0129! I agree with you that this could apply to more subpackages and so the discovery util could be moved into a more generic place.

Each subpackage will have their own exception list though so it could get a bit messy. For now I think this is good, but happy to consider a proposed refactor