Adaptive weights

[mariusmerkle]

Hi, @lululxvi

say we have several terms in the loss function, e.g. PDE loss, boundary loss, initial loss. Then we can impose fixed weights on the loss function with loss_weights in

model.compile("adam", lr = 1e-3, loss_weights = [1, 10, 10])

Is there a way to make these weights dynamic such that they counteract the issue of gradient imbalance between different magnitudes of gradients of different loss terms? As far as I know, there would be two possibilities in theory:

• adaptive weights to overcome gradient pathologies (https://arxiv.org/pdf/2001.04536.pdf, Wang et al.)
• Neural Tangent Kernel (https://arxiv.org/pdf/2007.14527.pdf, Wang et al.)

1. Is this possible to implement in DeepXDE?/Does there already exist a way to use adaptive weights in DeepXDE?
2. What's your experience on both fixed and adaptive weights in the loss function? Do adaptive weights accelerate convergence in your experience?

[lululxvi]

1. To do this, you need to either modify the source code, or implement a callback to change the weights during training.
2. Based on my experience, fixed and adaptive weights have similar effects. As you can see in the papers you mentioned, the adaptive weights quickly converge to a fixed number, and thus fixed weights are basically sufficient. Also, it is recommended to use hard constraints for BC/IC, see FAQ.