Update docstrings in upsample.jl, recurrent.jl, and normalise.jl

[Saransh-cpp]

Added missing docstrings (in the manual), updated the existing ones, and added doctests in the following files -

• normalise.jl
• recurrent.jl
• upsample.jl

PR Checklist

• Tests are added
• Entry in NEWS.md
• Documentation, if applicable

[codecov-commenter]

Codecov Report

Merging #1995 (2ab42cd) into master (0b01b77) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1995   +/-   ##
=======================================
  Coverage   87.10%   87.10%           
=======================================
  Files          20       20           
  Lines        1528     1528           
=======================================
  Hits         1331     1331           
  Misses        197      197           

Impacted Files	Coverage Δ
src/layers/normalise.jl	88.81% <ø> (ø)
src/layers/recurrent.jl	90.00% <ø> (ø)
src/layers/upsample.jl	69.56% <ø> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b01b77...2ab42cd. Read the comment docs.

[Saransh-cpp]

I tried a lot of filters, but I can't get the doctests for the Dropout layer to pass. The primary cause for this failure is the presence of 0.0 and -0.0 in the output, which the doctests can't handle.

For example, a regex pattern available here - https://regex101.com/r/1KtxKj/1 - correctly handles the 0.0 and -0.0 case, but it fails while running the doctests.

I have converted the jldoctest block to a julia block for the moment, but I will revert it back if I find something.

[Saransh-cpp]

Additionally, should the dropout function stay in the manual or should it be removed (internal API?)? If it stays, should I add examples for it too?

[ToucheSir]

Instead of trying to match the output, you can add a check that the number of zeros is equal to the dropout rate +- some margin of error.

I just checked the docs for dropout and it explicitly guides users towards Dropout, so additional examples are not necessary.

[Saransh-cpp]

Thank you for the review, @ToucheSir!

These should be unprefixed and imported from Statistics if not already.

Should I import Statistics in every doctest or should I import it only in the first doctest and group the other doctests together?

Instead of using comments and having to filter out the random numbers, why not compare directly against the thing in the comment? e.g: std(y[:, :, 1, 1]) \approx 1 && std(y[:, :, 1, 1]) \notapprox std(xs[:, :, 1, 1]) (can't find the symbols right now)

Yes, this would make the doctests much cleaner. However, I could not use the \approx symbol here due to the default low tolerance. Should I use the isapprox function with atol = some quantity (or perhaps rtol) here?

For this, I think it would actually be good to feed a fixed input through and display how the output changes because the operation is not terribly intuitive.

I did this initially (with random values, I really need to stop using random values), but then the extra vertical space taken changed my mind. I'll revert it back!

[ToucheSir]

It would be interesting to set the weight and bias to fixed values (even [1.0f0] might work, but I'm sure you could come up with something better) and show deterministic outputs much like you have for the accum example above. Really hammer home the iterative nature of the RNN interface

[Saransh-cpp]

I tried initializing an RNN with weights by passing them in through the RNNCell's constructor, but for some reason I cannot get it to work. Could you please let me know how can I set the weights to a fixed value?

[ToucheSir]

Can you share what you tried? That should work.

[Saransh-cpp]

I tried everything again today and it works 😅

julia> r = Flux.RNNCell(σ, zeros(1,1), zeros(1,1), zeros(1,1), zeros(1,1));
RNNCell(1 => 1, σ)

julia> y = Flux.Recur(r, ones(1,1));

julia> y.state
1×1 Matrix{Float64}:
 1.0

julia> y(ones(1,1))  # σ(0)
1×1 Matrix{Float64}:
 0.5

julia> y.state
1×1 Matrix{Float64}:
 0.5

julia> Flux.reset!(y)
1×1 Matrix{Float64}:
 0.0

julia> y.state
1×1 Matrix{Float64}:
 0.0

I am not sure if users are supposed to access these constructors like this? If they are then I can probably add this example to the doctests.

[ToucheSir]

Maybe just a comment that usually one does not create a RNN this way, but this is showing how data and hidden states are propagated?

[Saransh-cpp]

Not as clearly as shown by the accum example. I looked up in the documentation but couldn't figure out how to use RNNCells to create an example similar to the one using accum. But, this example does show that reset! resets the state to state0 of RNNCell rather than the initial state passed to Recur.

[ToucheSir]

You can choose a slightly easier to interpret activation function for the cell. I think sigmoid is a little hard to visualize, but there may be something better than just identity.

[Saransh-cpp]

relu might look good here. I have updated the example to use the weights instead of just setting them to 0 -

julia> r = Flux.RNNCell(relu, ones(1,1), zeros(1,1), ones(1,1), zeros(1,1));

julia> y = Flux.Recur(r, ones(1,1));

julia> y.state
1×1 Matrix{Float64}:
 1.0

julia> y(ones(1,1))  # relu(1*1 + 1)
1×1 Matrix{Float64}:
 2.0

julia> y.state
1×1 Matrix{Float64}:
 2.0

julia> Flux.reset!(y)
1×1 Matrix{Float64}:
 0.0

julia> y.state
1×1 Matrix{Float64}:
 0.0

[ToucheSir]

Looks good!

[ToucheSir]

Failure on Nightly is from ComponentArrays and can be safely ignored. Thanks for your patient work on this @Saransh-cpp !

[Saransh-cpp]

Thank you for all the constructive reviews!🎉🎉