-
-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NewRecur experimental interface #11
Conversation
Codecov ReportPatch coverage:
❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more. Additional details and impacted files@@ Coverage Diff @@
## master #11 +/- ##
==========================================
+ Coverage 76.87% 78.64% +1.77%
==========================================
Files 5 6 +1
Lines 147 206 +59
==========================================
+ Hits 113 162 +49
- Misses 34 44 +10
☔ View full report in Codecov by Sentry. |
Using I think something like |
I think we want both! Those who don't need the carry value or want to keep the Flux models are just plain callables property can deal with the added complexity of adding container layers. All other ML libraries require defining custom layers for this same purpose, so at least we can claim a reduction in boilerplate there. Those who care about the cleanliness of the model structure or feel this is too onerous can use the |
Ah hah! I understand now. I was thinking about it in terms of "one interface to rule them all", but manyboptions is much better. Likely we can combine a lot of the functionality here with the nonmutating recur. These functions are pretty much direct copies of the work done in the other PR. |
I decided to allow a vector of numbers and assume batch=1 and time=1 for inference, but maybe this is still too ambiguous and we should just can that all-together. I think we should just enforce blocking for this interface, instead of having both vector of arrays + blocks, but maybe that's being too restrictive. Any thoughts? I can support vector of arrays, but we really should fix |
Errors in 1.6 are confusing. Not sure why the exception is DimensionMismatch only for 1.6. I'll fix these when finalizing the interface. Next iteration:
|
I don't think the vector of arrays path needs |
Ok. I pushed most of the functionality of NewRecur to a scan function which is a bit more agnostic. Basically, I'm only enforcing the interface restrictions (i.e. The two scan functions ( |
Yes. You are correct. The issue is eachlastdim in Flux currently returns a vector of arrays when in a gradient environment vs a generator in a non-gradient environment (has to do w/ the collect in the rrule but not the base function). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apologies for the delayed review, wanted to sit down with and properly read through this one. Other than my questions about returning the carry and the implementation of scan_full
, the rest just needs a formatting pass.
src/new_recur.jl
Outdated
""" | ||
function scan_full(func, init_carry, xs::AbstractVector{<:AbstractArray}) | ||
# xs = Flux.eachlastdim(x_block) | ||
x_init, x_rest = Iterators.peel(xs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
x_init, x_rest = Iterators.peel(xs) | |
x_init = first(xs) |
Since x_rest
isn't used. Or is peel
more efficient in the contexts we care about?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We use x_rest in the foldl call to go through the rest of the sequence.
The reason peel was chosen here was not efficiency, but oddly gradient related. If instead I did
x_init = first(xs)
x_rest = xs[begin+1:end]
the resulting gradients were wrong. I've since documented this in the function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That seems very concerning. Do you have a MWE?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, no mwe. Commenting out the peel and replacing with the code block, I haven't been able to figure out where the gradients are going wrong here. Only know the tests fail.
src/new_recur.jl
Outdated
(carry, out_) = func(init_carry, x_init) | ||
|
||
init = (typeof(out_)[out_], carry) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also didn't quite understand why this is using carry
instead of init_carry
, can you walk me through that? AIUI this requires xs
to be non-empty
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are correct. xs is currently assumed to be non-empty. I've played around with only using foldl for this interface, but honestly I can't figure it out (i.e. Zygote hates it). I'm not sure why, and the error message is unfortunate. For this first pass, I think we should just do this, and then maybe bisect this choice and try and figure out why Zygote is not liking the other version in another issue/PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My understanding is that Lux handles this with less code. Can we do the same? I don't think a dedicated rule is required for foldl like theye have, however.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, so if we try and use a similar implementation to the current Lux implementation it would look like the below.
function scan_full(func, init_carry, xs::AbstractVector{<:AbstractArray})
function __recurrence_op(::Nothing, input)
carry, out = func(init_carry, input)
return carry, [out]
end
# recurrence operation used in the fold. Takes the state of the
# folde and the next input, returns the new state.
function __recurrence_op((carry, outputs), input)
carry, out = func(carry, input)
return carry, vcat(outputs, [out])
end
# Fold left to right.
foldl(__recurrence_op, xs; init=nothing)
end
Unfortunately, gradients won't compile afaict. I'm not sure why we don't get the same behavior as Lux.jl. Also preemptively Lux's foldl_init
just calls an inline for foldl
. Using foldl_init
also fails in a similar way.
Here is the stack trace when running the tests. I have no idea what to takeaway from this error, and any help would be great.
Stack Trace
gradients-implicit: Error During Test at /Users/Matt/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:30
Got exception outside of a @test
MethodError: Cannot `convert` an object of type
Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Matrix{Float32}, Vector{Matrix{Float32}}},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Matrix{Float32}, Vector{Matrix{Float32}}},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2,1,Zygote.Context{true},Array{Matrix{Float32},1}}},Zygote.var"#2017#back#200"{typeof(identity)},Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}} to an object of type
Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Nothing,SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Nothing,SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2,1,Zygote.Context{true},Array{Float32,2}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}},Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}},Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2017#back#200"{typeof(identity)}}}}
Closest candidates are:
convert(::Type{T}, ::T) where T
@ Base Base.jl:64
Stacktrace:
[1] cvt1
@ ./essentials.jl:418 [inlined]
[2] ntuple
@ ./ntuple.jl:49 [inlined]
[3] convert(#unused#::Type{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}}})
@ Base ./essentials.jl:419
[4] setindex!(A::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}}}, i1::Int64)
@ Base ./array.jl:969
[5] _accumulate1!(op::ChainRules.var"#1715#1717"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, v1::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dim::Int64)
@ Base ./accumulate.jl:432
[6] _accumulate!(op::ChainRules.var"#1715#1717"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dims::Nothing, init::Some{Tuple{Nothing, Nothing}})
@ Base ./accumulate.jl:361
[7] accumulate!(op::Function, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}; dims::Nothing, kw::Base.Pairs{Symbol, Tuple{Nothing, Nothing}, Tuple{Symbol}, NamedTuple{(:init,), Tuple{Tuple{Nothing, Nothing}}}})
@ Base ./accumulate.jl:342
[8] accumulate!
@ ./accumulate.jl:337 [inlined]
[9] #accumulate#872
@ ./accumulate.jl:285 [inlined]
[10] accumulate
@ ./accumulate.jl:272 [inlined]
[11] #rrule#1714
@ ~/.julia/packages/ChainRules/U21Ei/src/rulesets/Base/mapreduce.jl:440 [inlined]
[12] rrule
@ ~/.julia/packages/ChainRules/U21Ei/src/rulesets/Base/mapreduce.jl:430 [inlined]
[13] chain_rrule_kw
@ ~/.julia/packages/Zygote/JeHtr/src/compiler/chainrules.jl:235 [inlined]
[14] macro expansion
@ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined]
[15] _pullback
@ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined]
[16] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:56 [inlined]
[17] _pullback(::Zygote.Context{true}, ::typeof(Fluxperimental.scan_full), ::Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, ::Matrix{Float32}, ::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
[18] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:70 [inlined]
[19] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:142 [inlined]
[20] _pullback(::Zygote.Context{true}, ::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, ::Matrix{Float32}, ::Array{Float32, 3})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
[21] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:131 [inlined]
[22] _pullback(ctx::Zygote.Context{true}, f::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, args::Array{Float32, 3})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
[23] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:52 [inlined]
[24] _pullback(::Zygote.Context{true}, ::var"#3#5"{Array{Float32, 3}, Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
[25] pullback(f::Function, ps::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:384
[26] withgradient(f::Function, args::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:132
[27] macro expansion
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:51 [inlined]
[28] macro expansion
@ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
[29] macro expansion
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:31 [inlined]
[30] macro expansion
@ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
[31] top-level scope
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:4
[32] include(fname::String)
@ Base.MainInclude ./client.jl:478
[33] macro expansion
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11 [inlined]
[34] macro expansion
@ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
[35] top-level scope
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11
[36] include(fname::String)
@ Base.MainInclude ./client.jl:478
[37] top-level scope
@ none:6
[38] eval
@ ./boot.jl:370 [inlined]
[39] exec_options(opts::Base.JLOptions)
@ Base ./client.jl:280
[40] _start()
@ Base ./client.jl:522
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Aha I think I know what this is. Long story short, ChainRules tries to store an array of pullbacks, but since both method overloads have slightly different captured variables, they result in different pullbacks. Now I know why Lux has custom rrules for this: they erase the type of the pullback array to avoid the conversion error.
Does the following change work?
function scan_full(func, init_carry, xs::AbstractVector{<:AbstractArray})
function __recurrence_op(::Tuple{Nothing, Nothing}, input)
carry, out = func(init_carry, input)
return carry, [out]
end
# recurrence operation used in the fold. Takes the state of the
function __recurrence_op((carry, outputs), input)
...
end
# Fold left to right.
foldl(__recurrence_op, xs; init=(nothing, nothing))
end
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What error do you get using the second one?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems similar to the previous error.
gradients-implicit: Error During Test at /Users/Matt/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:30
Got exception outside of a @test
MethodError: Cannot `convert` an object of type
Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Matrix{Float32},Vector{Matrix{Float32}}},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Matrix{Float32},Vector{Matrix{Float32}}},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2,1,Zygote.Context{true},Array{Matrix{Float32},1}}},Zygote.var"#2017#back#200"{typeof(identity)},Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}} to an object of type
Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Nothing,Nothing},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Nothing,Nothing},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2,1,Zygote.Context{true},Array{Float32,2}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}},Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}},Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2017#back#200"{typeof(identity)}}}}
Closest candidates are:
convert(::Type{T}, ::T) where T
@ Base Base.jl:64
Stacktrace:
[1] cvt1
@ ./essentials.jl:418 [inlined]
[2] ntuple
@ ./ntuple.jl:49 [inlined]
[3] convert(#unused#::Type{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}}})
@ Base ./essentials.jl:419
[4] setindex!(A::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}}}, i1::Int64)
@ Base ./array.jl:969
[5] _accumulate1!(op::ChainRules.var"#1715#1717"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, v1::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dim::Int64)
@ Base ./accumulate.jl:432
[6] _accumulate!(op::ChainRules.var"#1715#1717"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dims::Nothing, init::Some{Tuple{Tuple{Nothing, Nothing}, Nothing}})
@ Base ./accumulate.jl:361
[7] accumulate!(op::Function, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}; dims::Nothing, kw::Base.Pairs{Symbol, Tuple{Tuple{Nothing, Nothing}, Nothing}, Tuple{Symbol}, NamedTuple{(:init,), Tuple{Tuple{Tuple{Nothing, Nothing}, Nothing}}}})
@ Base ./accumulate.jl:342
[8] accumulate!
@ ./accumulate.jl:337 [inlined]
[9] #accumulate#872
@ ./accumulate.jl:285 [inlined]
[10] accumulate
@ ./accumulate.jl:272 [inlined]
[11] #rrule#1714
@ ~/.julia/packages/ChainRules/U21Ei/src/rulesets/Base/mapreduce.jl:440 [inlined]
[12] rrule
@ ~/.julia/packages/ChainRules/U21Ei/src/rulesets/Base/mapreduce.jl:430 [inlined]
[13] chain_rrule_kw
@ ~/.julia/packages/Zygote/JeHtr/src/compiler/chainrules.jl:235 [inlined]
[14] macro expansion
@ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined]
[15] _pullback
@ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined]
[16] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:50 [inlined]
[17] _pullback(::Zygote.Context{true}, ::typeof(Fluxperimental.scan_full), ::Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, ::Matrix{Float32}, ::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
[18] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:106 [inlined]
[19] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:178 [inlined]
[20] _pullback(::Zygote.Context{true}, ::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, ::Matrix{Float32}, ::Array{Float32, 3})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
[21] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:167 [inlined]
[22] _pullback(ctx::Zygote.Context{true}, f::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, args::Array{Float32, 3})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
[23] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:52 [inlined]
[24] _pullback(::Zygote.Context{true}, ::var"#3#5"{Array{Float32, 3}, Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
[25] pullback(f::Function, ps::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:384
[26] withgradient(f::Function, args::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:132
[27] macro expansion
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:51 [inlined]
[28] macro expansion
@ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
[29] macro expansion
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:31 [inlined]
[30] macro expansion
@ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
[31] top-level scope
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:4
[32] include(fname::String)
@ Base.MainInclude ./client.jl:478
[33] macro expansion
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11 [inlined]
[34] macro expansion
@ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
[35] top-level scope
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11
[36] include(fname::String)
@ Base.MainInclude ./client.jl:478
[37] top-level scope
@ none:6
[38] eval
@ ./boot.jl:370 [inlined]
[39] exec_options(opts::Base.JLOptions)
@ Base ./client.jl:280
[40] _start()
@ Base ./client.jl:522
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/rulesets/Base/mapreduce.jl:440
This is https://github.com/JuliaDiff/ChainRules.jl/blob/main/src/rulesets/Base/mapreduce.jl#L440 which is part of the rule for foldl
. Which I wrote and is a bit of a monster.
If it's easy to do so, trying with JuliaDiff/ChainRules.jl#569 might be worthwhile This re-writes the rule to solve various problems, esp. to do with init
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using mapfoldl_impl
would work well enough, here's a rough sketch:
function scan_full(func, init_carry, xs::AbstractVector{<:AbstractArray})
# Recurrence operation used in the fold. Takes the state of the
# fold and the next input, returns the new state.
function recurrence_op((carry, outputs), input)
carry, out = func(carry, input)
return carry, vcat(outputs, [out])
end
# Fold left to right.
return Base.mapfoldl_impl(identity, recurrence_op, (init_carry, empty(xs)), xs)
end
function rrule(
config::RuleConfig{>:HasReverseMode}, ::typeof(Base.mapfoldl_impl), ::typeof(identity), op::G, init, x::Union{AbstractArray, Tuple};
) where {G}
hobbits = accumulate(x; init=(init, nothing)) do (a, _), b
c, back = rrule_via_ad(config, op, a, b)
end
y = first(last(hobbits))
axe = axes(x)
project = ProjectTo(x)
function unfoldl(dy)
trio = accumulate(Iterators.reverse(hobbits); init=(0, dy, 0)) do (_, dc, _), (_, back)
ds, da, db = back(dc)
end
dop = sum(first, trio)
dx = map(last, Iterators.reverse(trio))
d_init = trio[end][2]
return (NoTangent(), NoTangent(), dop, d_init, project(reshape(dx, axe)))
end
return y, unfoldl
end
I think the problem is that the rrule which takes an Array in that PR doesn't provide a gradient for the init value. I don't think it would be too hard to add (see simplified method above) but I may be missing something.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm getting a similar error still, and it is in the line hobbits = accumulate...
. I'm not too sure how to debug this, as it is a convert issue when accumulating the gradient. When I get a bit of time I will try and debug a bit more, but I'm at a loss with the current error, so any help would be appreciated.
gradients-implicit: Error During Test at /Users/Matt/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:30
Got exception outside of a @test
MethodError: Cannot `convert` an object of type
Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}}}}} to an object of type
Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}
Closest candidates are:
convert(::Type{T}, ::T) where T
@ Base Base.jl:64
Stacktrace:
[1] cvt1
@ ./essentials.jl:418 [inlined]
[2] ntuple
@ ./ntuple.jl:49 [inlined]
[3] convert(#unused#::Type{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}}}}}})
@ Base ./essentials.jl:419
[4] setindex!(A::Vector{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}}}}}}, i1::Int64)
@ Base ./array.jl:969
[5] _accumulate1!(op::Fluxperimental.var"#36#38"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, v1::Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{Any}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dim::Int64)
@ Base ./accumulate.jl:432
[6] _accumulate!(op::Fluxperimental.var"#36#38"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dims::Nothing, init::Some{Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}})
@ Base ./accumulate.jl:361
[7] accumulate!(op::Function, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}; dims::Nothing, kw::Base.Pairs{Symbol, Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}, Tuple{Symbol}, NamedTuple{(:init,), Tuple{Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}}}})
@ Base ./accumulate.jl:342
[8] accumulate(op::Function, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}; dims::Nothing, kw::Base.Pairs{Symbol, Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}, Tuple{Symbol}, NamedTuple{(:init,), Tuple{Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}}}})
@ Base ./accumulate.jl:285
[9] accumulate
@ ./accumulate.jl:272 [inlined]
[10] rrule(config::Zygote.ZygoteRuleConfig{Zygote.Context{true}}, ::typeof(Base.mapfoldl_impl), ::typeof(identity), op::Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, init::Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, x::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}})
@ Fluxperimental ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:48
[11] chain_rrule(::Zygote.ZygoteRuleConfig{Zygote.Context{true}}, ::Function, ::Function, ::Vararg{Any})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/chainrules.jl:223
[12] macro expansion
@ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined]
[13] _pullback(::Zygote.Context{true}, ::typeof(Base.mapfoldl_impl), ::typeof(identity), ::Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, ::Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, ::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101
[14] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:42 [inlined]
[15] _pullback(::Zygote.Context{true}, ::typeof(Fluxperimental.scan_full), ::Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, ::Matrix{Float32}, ::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
[16] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:137 [inlined]
[17] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:209 [inlined]
[18] _pullback(::Zygote.Context{true}, ::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, ::Matrix{Float32}, ::Array{Float32, 3})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
[19] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:198 [inlined]
[20] _pullback(ctx::Zygote.Context{true}, f::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, args::Array{Float32, 3})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
[21] _pullback
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:52 [inlined]
[22] _pullback(::Zygote.Context{true}, ::var"#3#5"{Array{Float32, 3}, Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0
[23] pullback(f::Function, ps::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:384
[24] withgradient(f::Function, args::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}})
@ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:132
[25] macro expansion
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:51 [inlined]
[26] macro expansion
@ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
[27] macro expansion
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:31 [inlined]
[28] macro expansion
@ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
[29] top-level scope
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:4
[30] include(fname::String)
@ Base.MainInclude ./client.jl:478
[31] macro expansion
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11 [inlined]
[32] macro expansion
@ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
[33] top-level scope
@ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11
[34] include(fname::String)
@ Base.MainInclude ./client.jl:478
[35] top-level scope
@ none:6
[36] eval
@ ./boot.jl:370 [inlined]
[37] exec_options(opts::Base.JLOptions)
@ Base ./client.jl:280
[38] _start()
@ Base ./client.jl:522
src/new_recur.jl
Outdated
(l::NewRecur)(init_carry, x_mat::AbstractMatrix) = MethodError("Matrix is ambiguous with NewRecur") | ||
(l::NewRecur)(init_carry, x_mat::AbstractVector{T}) where {T<:Number} = MethodError("Vector is ambiguous with NewRecur") | ||
|
||
(l::NewRecur)(xs) = l(l.cell.state0, xs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(l::NewRecur)(xs) = l(l.cell.state0, xs) | |
(l::NewRecur)(xs::AbstractArray) = l(l.cell.state0, xs) |
For your consideration
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Likely this would be a good idea. But I didn't restrict this in this initial pass.
src/new_recur.jl
Outdated
function (l::NewRecur{true})(init_carry, | ||
xs,) | ||
|
||
results = scan_full(l.cell, init_carry, xs) | ||
|
||
h = results[2] | ||
sze = size(h[1]) | ||
reshape(reduce(hcat, h), sze[1], sze[2], length(h)) | ||
end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
function (l::NewRecur{true})(init_carry, | |
xs,) | |
results = scan_full(l.cell, init_carry, xs) | |
h = results[2] | |
sze = size(h[1]) | |
reshape(reduce(hcat, h), sze[1], sze[2], length(h)) | |
end | |
function (l::NewRecur{true})(init_carry, xs) | |
results = scan_full(l.cell, init_carry, xs) | |
return results[1], stack(results[2]) | |
end |
Similar story here. stack
vs reduce(hcat
are more or less equally efficient so feel free to use either.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed this to stack like you suggested.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't realize stack
is not available in 1.6. This function was added to 1.9. Are we targeting only 1.9?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's available via Compat.jl, which Flux already has as a transitive dep so there's zero additional import overhead.
Here's a possibly crazy idea for handling carry values: can we split the NewRecur call function into one which takes a carry and returns a carry, and one which takes no carry and returns no carry? I suspect the latter will cover most use cases while avoiding any additional model complexity with container layers. Those who need the former are probably power users who would be more comfortable with advanced use of container layers and/or defining their own layers. |
Sorry that I haven't been following these discussions closely, but I'm catching up now pre-0.14.
I prefer this option. To be honest, I don't really get the "optimization" of not returning everything. The performance difference is not doing a reshape, so it boils down to convenience within a |
I think that is a great compromise. We could also do a parameter option (like with I do think it might be difficult for new users to understand this particular overload though. I.e. changing the function arguments changes the return type, but that is generally the case in Julia so likely ok. @darsnack All good! Just to be clear, you mean for the carry correct? We still will use a flag type parameter to determine whether to return the whole sequence or not (this is different than the carry question). Any reason not to do that (i.e. @ToucheSir Thanks for the review! This definitely needed a pass from you, and a lot of these decisions came up in the other recur interface for the new chain. So documenting them here would be good. The delay actually worked for me due to job interviews 😆. Working on this today. |
Ok. Took another pass and made almost all of your changes @ToucheSir. I didn't use the edits you made directly because there was some specific issues that needed to be resolved throughout to make the changes work. I've not resolved the conversations for the specific changes that I either didn't do/which needed more info/conversation. And I reopened the stack due to it only being available on 1.9. |
Ok. Cleaned up the code a bit and fixed the gradients. We are failing on 1.6, but because Flux now requires 1.9, I'm not sure that is an issue. Unfortunately, |
I think adding Compat as a dep and using stack from there instead of Base should be enough. Would be fine with bumping the min Julia compat as well though. |
Likely the right solution would be to up the bounds for julia, but I just added Compat as a dependency. This project will probably need it anyway as time goes on. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you mind removing all the extraneous newlines in {src,test}/new_recur.jl
? Otherwise LGTM!
Done. |
Adding NewRecur as proposed in FluxML/Flux.jl#2258. This does indeed fix the gradient issues in explicit mode.
Added Features
NewRecur{SEQ}
which implements the interface proposed by @ToucheSir.Issues:
Thoughts:
Chain
instead of forcing the user to manage the network structure.PR Checklist
[ ] Documentation, if applicable