[Feature] Support dynamic EMA #261
Conversation
| momentum_cfg) | ||
| self.momentum_policy = momentum_policy | ||
| if momentum_policy != 'fixed': | ||
| assert hasattr( |
There was a problem hiding this comment.
I suggest that we can add a class attribute called _supported_momentum_policy. Use instance attributes' names to check the policy may not be safe.
There was a problem hiding this comment.
True, may be we can register interp_func and momentum_updater in this class?
|
|
||
| @staticmethod | ||
| def lerp(a, b, momentum=0.999, momentum_nontrainable=0., trainable=True): | ||
| m = momentum if trainable else momentum_nontrainable | ||
| return a + (b - a) * m | ||
|
|
||
| @staticmethod | ||
| def rampup(runner, ema_kimg=10, ema_rampup=0.05, batch_size=4, eps=1e-8): |
There was a problem hiding this comment.
Maybe we should change the naming template of momentum update policy as {POLICY}_momentum_updater or get_{POLICY}_momentum.
Because we had already used the name of interpolating policy (e.g. lerp) as the function name. Using momentum update policy as function name may lead to confusion.
There was a problem hiding this comment.
may be we can register interp_func and momentum_updater in this class? Keep these functions in two separate function dictionaries.
* support update ema momentum * complete docstring * fix lint * fix as comment
* tiny fix * Tiny Fix, add limited_gpu. * Tiny Fix * Tiny Fix Co-authored-by: liyinshuo <liyinshuo@sensetime.com>
This PR modify ema hook, refer to https://github.com/NVlabs/stylegan3.
EMAHook.