Issue #96

[M1ngXU]

See Issue #96

[M1ngXU]

The Module::forward is quite inefficient with 3 allocations, so the old residual_add-connection is now residual_add. Also, I have no idea about the numpy stuff, so I am not sure how to fix the last numpy test (comments in code).

[coreylowman]

Thanks for the contribution!

[coreylowman]

I like the idea of having both Residual & ResidualAdd. I imagine most people will probably use ResidualAdd, so let's rename them

• Residual does f(x) + x
• GeneralizedResidual does f(x) + r(x)

[M1ngXU]

What about SkipConnection instead of GeneralizedResidual?

[M1ngXU]

ok, after re-reading some wiki pages i think that i understand it better now - but shouldn't the gradients of F and R have to be summed AFTER the backprop of F and R (so the addition would have to be pushed to the tape first?) or am I not understanding binary_map ?

[coreylowman]

All the operations on the tape are executed in reverse, so for this function the add/binary_map backward op will be executed, then the f(x) backward op is executed using the gradient from add(), and then the r(x) backward ops using gradient from add(). Does that help?

Let's keep both Residual and SkipConnection, and then GeneralizedResidual. SkipConnection can just be a type alias (type SkipConnection<F, R> = Residual<F, R>;

[M1ngXU]

do you have an idea why the numpy test fails?

[M1ngXU]

add_grad + f_grad + r_grad

but if the add(y, &x) is executed before 2.a in the backprop, how can f_grad be fetched from gradients to calculate x's gradients for 1?

[coreylowman]

The three are summed together independently. After add's backprop x's gradient is only 1/3 complete (add_grad). Once f's backward op is executed then x is 2/3 complete (add_grad + f_grad), then finally after r's backward op x is complete (add_grad + f_grad + r_grad).

It works because x is input for add, f, and r, not the result. Gradients only need to be fully "complete" for backward ops in which they are the result.

[M1ngXU]

but for the last backward op the gradient of x = r_grad + f_grad. in which line is this done? i can only find the backprops for r_grad and f_grad, but not their addition

[coreylowman]

it depends on what f & r are. for binary map the call to addmul does rhs_grad += rhs_deriv * result_grad, so if rhs_grad is x's grad that's where part of the accumulation happens.

If F and R were both using map for instance then both accumulations would happen in map (where x's gradient is g)

[M1ngXU]

Linear<10, 5>

i mean that linear layer, what gradients will it use? it will be lhs_grad, won't it? but when is rhs_grad added to lhs_grad?

[M1ngXU]

hmmm, file deleted and new file oof

[M1ngXU]

should be ready for merge now

[M1ngXU]

finally the tests all passed 😅

[M1ngXU]

with the new GeneralizedResidual, the Residual struct can be removed, if we add a Linear-Activation, so f(x) = x, and then type Residual<F> = GeneralizedResidual<F, Linear>

[coreylowman]

@M1ngXU this looks good to me for merging - can you update or rebase your branch to current master (e.g. the dropout change was already merged into main)

[M1ngXU]

@M1ngXU this looks good to me for merging - can you update or rebase your branch to current master (e.g. the dropout change was already merged into main)

how exactly do i do that? 😅

[coreylowman]

@M1ngXU this looks good to me for merging - can you update or rebase your branch to current master (e.g. the dropout change was already merged into main)

how exactly do i do that? 😅

This article should walk you through it https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[M1ngXU]

@M1ngXU this looks good to me for merging - can you update or rebase your branch to current master (e.g. the dropout change was already merged into main)

how exactly do i do that? 😅

This article should walk you through it https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

ok, should work now. i thought that i already tried that, but i appearently synced main and not BetterResidual.

[M1ngXU]

with the new GeneralizedResidual, the Residual struct can be removed, if we add a Linear-Activation, so f(x) = x, and then type Residual<F> = GeneralizedResidual<F, Linear>

@coreylowman how about this?

[coreylowman]

with the new GeneralizedResidual, the Residual struct can be removed, if we add a Linear-Activation, so f(x) = x, and then type Residual<F> = GeneralizedResidual<F, Linear>

I like the idea about moving Residual to use GeneralizedResidual, but I think we'd need something like Identity (which just returns x in the forward pass) instead of Linear for R.

[M1ngXU]

with the new GeneralizedResidual, the Residual struct can be removed, if we add a Linear-Activation, so f(x) = x, and then type Residual<F> = GeneralizedResidual<F, Linear>

I like the idea about moving Residual to use GeneralizedResidual, but I think we'd need something like Identity (which just returns x in the forward pass) instead of Linear for R.

the name Identity makes more sense