add ClipValue and ClipNorm

[AStupidBear]

No description provided.

[bhvieira]

Keras has these, good to have it on Flux as well

[AStupidBear]

@bhvieira The docstrings are simpified now.

[DhairyaLGandhi]

Please add them to the tests as well

[mcabbott]

Should both of these be called clipping, given that one clamps (only components outside range) and the other scales (all components)? Also they act on the gradient not the value. Maybe ClampGrad and, I don't know, LimitGradNorm, BoundGrad?

Also: is the norm of the whole array always what you want? If this is some batch of observations, then it may make more sense to scale each of them by its own norm. In which case this would need to take dims somehow.

[HenriDeh]

norm uses scalar getindex. This throws an error when working with CuArrays. eg:

norm(cu(rand(10,10)), 2)

The issue comes from CuArrays. However, norm(cu(rand(10,10))) is implemented.

[AStupidBear]

Should both of these be called clipping, given that one clamps (only components outside range) and the other scales (all components)? Also they act on the gradient not the value. Maybe ClampGrad and, I don't know, LimitGradNorm, BoundGrad?

Also: is the norm of the whole array always what you want? If this is some batch of observations, then it may make more sense to scale each of them by its own norm. In which case this would need to take dims somehow.

Should both of these be called clipping, given that one clamps (only components outside range) and the other scales (all components)? Also they act on the gradient not the value. Maybe ClampGrad and, I don't know, LimitGradNorm, BoundGrad?

Also: is the norm of the whole array always what you want? If this is some batch of observations, then it may make more sense to scale each of them by its own norm. In which case this would need to take dims somehow.

I think it's better to just follow Keras's convention. Gradients are not batched, so...

[AStupidBear]

norm uses scalar getindex. This throws an error when working with CuArrays. eg:

norm(cu(rand(10,10)), 2)

The issue comes from CuArrays. However, norm(cu(rand(10,10))) is implemented.

@HenriDeh Thanks for pointing that out.

[mcabbott]

Gradients are not batched, so...
Gradients are not batched, so...

Oh sorry, this only applies to parameter updates, not to gradients in general. Which is obvious in the code, apply!, but not in the present docstrings.

[AStupidBear]

Since this PR is just a simple feature add-on, how about merging this?

[DhairyaLGandhi]

bors try

[bors]

try

Build succeeded:

• ci/gitlab/gitlab.com

[CarloLucibello]

We need usage examples in the docstrings and we should add the two to the docs/

[CarloLucibello]

this seems good to go, thanks

bors r+

[bors]

Build succeeded:

• ci/gitlab/gitlab.com