Feature request: Modifying Dense Layer to accommodate kernel/bias constraints and kernel/bias regularisation

[yewalenikhil65]

Hi,
I feel Dense layer should be more armed with arguments like kernel/weights constraints, bias constraints, kernel/weights regularisation and bias regularisation as is available in Tensorflow.
Weight constraints and bias constraints could help in avoiding overfitting

tf.keras.layers.Dense(
    units, activation=None, use_bias=True, kernel_initializer='glorot_uniform',
    bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None,
    activity_regularizer=None, kernel_constraint=None, bias_constraint=None,
    **kwargs)

Some well-known weight/bias constraints include NonNeg (to make weights non-negative)
MaxNorm, MinMaxNorm and UnitNorm as documented at
https://keras.io/api/layers/constraints/

[jeremiedb]

This is just a personal impression, but my understanding is that the Flux philosophy would be to handle such options at the loss/optimization level. For example, the examples presented here https://fluxml.ai/Flux.jl/stable/models/regularisation/ shows how ad hoc constraints on parameters could be applied. Also, https://fluxml.ai/Flux.jl/stable/models/advanced/ shows how constraints could be applied through params freezing.

By doing so, I think it helps keeping the control flexible and applicable to any operators without the need to load numerous arguments to each of the operator, which is a plus to the Flux experience in my opinion.

[yewalenikhil65]

Hi @jeremiedb
I think you are right.
But does this ad-hoc constraint using params (as mentioned in documentation you Linked) is applied during the whole training process when we use Flux.train! ? I think not.

I had little difficulty in understanding this

[CarloLucibello]

You can pass a callback to the train! function or define your custom training loop
https://fluxml.ai/Flux.jl/stable/training/training/#Custom-Training-loops-1

Typically constraints are implemented by contracting the weights to the constrained space after each update, e.g.

for p in params(model)
    p .= clamp(p, -2, 2)
end

or reparametrizing the weights.
The former is what keras' constraints do. From the page you linked: " They are per-variable projection functions applied to the target variable after each gradient update (when using fit())."

In the latter case instead, you have to define your own layer.
See weight norm #1005 for an incomplete attempt to extend the reparametrization to all layers.

[yewalenikhil65]

@CarloLucibello
Thank you for your suggestion.
Did you mean following way of passing a callback function ?

ps = Flux.params(model)
cb = function()	#callback function
	ps[1] .= abs.(ps[1])  # to , "say" only consider positive or absolute values of first layer

       # or clamping in 0.0 and 1.0 interval
       ps[1] .= clamp.(ps[1], 0.0 ,1.0)
end
cb()
@epochs args.epochs Flux.train!( loss, ps, train_data, opt, cb = cb)