Temporarily suspend one optimizer

[turian]

🚀 Feature

It should be possible to temporarily suspend one optimizer.

Motivation

When training a GAN, if the generator or discriminator becomes too good, one might want to suspend that module's optimizer until the other module catches up.

Pitch

There should be a simple clean way to disable and re-enable a particular optimizer, so that its training steps are completely skipped until they are re-enabled.

Alternatives

• Tried returning a simple loss, but got an error because the gradient couldn't backprop.
• Tried disabling that module's gradients, but also got an error.
• Considered zero'ing the gradients manually, but that seems really fiddly and gross.

Additional context

In general, allowing finer control over how multiple optimizers works would be beneficial in many settings.

[ananthsub]

@turian https://pytorch-lightning.readthedocs.io/en/latest/common/optimizers.html#manual-optimization should offer finer control over optimizer steps - does this work for your use case?

[turian]

@ananthsub Thanks. It also appears that if I return loss = None it toggles the optimizer: #5983