NewRecur experimental interface #11

mkschleg · 2023-06-26T00:06:36Z

Adding NewRecur as proposed in FluxML/Flux.jl#2258. This does indeed fix the gradient issues in explicit mode.

Added Features

NewRecur{SEQ} which implements the interface proposed by @ToucheSir.
Tested gradients of the NewRecur.

Issues:

Have no solution to return the state of the network to the user currently. Might not be simple, and I don't think forcing users to wrap everything in several parallel streams is a viable option interface wise.
Have not tested with passing in carry to the cell, but it should work pretty easily.

Thoughts:

I prefer the interface worked on for the new chain interface we worked on several months ago to this. I think that provides a way for us to return the carry a bit easier, but there might be problems still there (see Adding non-mutating recur for the new chain interface. #7).
I think taking the state out of the network is necessary, but if we do this we should go all in on the idea, and incorporate the change into Chain instead of forcing the user to manage the network structure.

PR Checklist

Tests are added
~~[ ] Documentation, if applicable~~

codecov-commenter · 2023-06-26T00:12:39Z

Codecov Report

Patch coverage: 83.05% and project coverage change: +1.77% 🎉

Comparison is base (dfda16c) 76.87% compared to head (52f3b7f) 78.64%.
Report is 2 commits behind head on master.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #11      +/-   ##
==========================================
+ Coverage   76.87%   78.64%   +1.77%     
==========================================
  Files           5        6       +1     
  Lines         147      206      +59     
==========================================
+ Hits          113      162      +49     
- Misses         34       44      +10

Files Changed	Coverage Δ
src/Fluxperimental.jl	`100.00% <ø> (ø)`
src/new_recur.jl	`83.05% <83.05%> (ø)`

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mkschleg · 2023-06-26T00:14:48Z

Using first or last if NewRecur returns both the carry and the output of the cell could be a viable option. (I opted here to just return the output for now). This still wouldn't solve the issue I discuss above about the user not having access to the carry over the sequence at all.

I think something like Chain(Parallel(encoder, identity), NewRecur{SEQ, CELL}, Parallel(decoder, identity)) would be extremely onerous. But if there is no better way w/o creating a tree of state and incorporate the pass through in Chain (similar to Lux.jl), then I'm not sure what to do.

ToucheSir · 2023-06-26T00:41:11Z

I prefer the interface worked on for the new chain interface we worked on several months ago to this. I think that provides a way for us to return the carry a bit easier, but there might be problems still there (see #7).

I think we want both! Those who don't need the carry value or want to keep the Flux models are just plain callables property can deal with the added complexity of adding container layers. All other ML libraries require defining custom layers for this same purpose, so at least we can claim a reduction in boilerplate there. Those who care about the cleanliness of the model structure or feel this is too onerous can use the apply interface.

mkschleg · 2023-06-26T01:00:22Z

Ah hah! I understand now. I was thinking about it in terms of "one interface to rule them all", but manyboptions is much better. Likely we can combine a lot of the functionality here with the nonmutating recur. These functions are pretty much direct copies of the work done in the other PR.

src/new_recur.jl

mkschleg · 2023-06-26T16:46:05Z

I decided to allow a vector of numbers and assume batch=1 and time=1 for inference, but maybe this is still too ambiguous and we should just can that all-together.

I think we should just enforce blocking for this interface, instead of having both vector of arrays + blocks, but maybe that's being too restrictive. Any thoughts? I can support vector of arrays, but we really should fix eachlastdim to make that interface less wonky (i.e. needing to support generators).

mkschleg · 2023-06-26T16:50:08Z

Errors in 1.6 are confusing. Not sure why the exception is DimensionMismatch only for 1.6. I'll fix these when finalizing the interface.

Next iteration:

Support arbitrary N dim arrays (with last dim assumed time).
Remove support for flat Vector of numbers (if we decide to do this).
Re-add support for vector of arrays (if we decide to do this).

ToucheSir · 2023-06-27T04:18:17Z

I don't think the vector of arrays path needs eachlastdim, just the blocked one? Maybe I'm missing something here. But yes, I'd be fine with disallowing a vector of numbers.

mkschleg · 2023-06-27T19:15:50Z

Ok. I pushed most of the functionality of NewRecur to a scan function which is a bit more agnostic. Basically, I'm only enforcing the interface restrictions (i.e. N>=3) in the NewRecur interface, but still allow for pass through of vector of arrays (although this pass through may have unattended consequences I'm not seeing).

The two scan functions (scan_partial, scan_full) can likely be put into NNLib. Although we might want to rethink the design of these. Likely could combine into a single function with different methods through dispatching on a helper struct, but that is a different design process to be thought through (but we should be able to easily cover all of jax.lax.scan's functionality).

mkschleg · 2023-06-27T19:20:54Z

I don't think the vector of arrays path needs eachlastdim, just the blocked one? Maybe I'm missing something here. But yes, I'd be fine with disallowing a vector of numbers.

Yes. You are correct. The issue is eachlastdim in Flux currently returns a vector of arrays when in a gradient environment vs a generator in a non-gradient environment (has to do w/ the collect in the rrule but not the base function).

ToucheSir

Apologies for the delayed review, wanted to sit down with and properly read through this one. Other than my questions about returning the carry and the implementation of scan_full, the rest just needs a formatting pass.

src/new_recur.jl

ToucheSir · 2023-07-08T15:14:00Z

src/new_recur.jl

+"""
+function scan_full(func, init_carry, xs::AbstractVector{<:AbstractArray})
+  # xs = Flux.eachlastdim(x_block)
+  x_init, x_rest = Iterators.peel(xs)


Suggested change

x_init, x_rest = Iterators.peel(xs)

x_init = first(xs)

Since x_rest isn't used. Or is peel more efficient in the contexts we care about?

We use x_rest in the foldl call to go through the rest of the sequence.

The reason peel was chosen here was not efficiency, but oddly gradient related. If instead I did

x_init = first(xs) x_rest = xs[begin+1:end]

the resulting gradients were wrong. I've since documented this in the function.

That seems very concerning. Do you have a MWE?

No, no mwe. Commenting out the peel and replacing with the code block, I haven't been able to figure out where the gradients are going wrong here. Only know the tests fail.

ToucheSir · 2023-07-08T18:20:13Z

src/new_recur.jl

+  (carry, out_) = func(init_carry, x_init)
+
+  init = (typeof(out_)[out_], carry)


I also didn't quite understand why this is using carry instead of init_carry, can you walk me through that? AIUI this requires xs to be non-empty

You are correct. xs is currently assumed to be non-empty. I've played around with only using foldl for this interface, but honestly I can't figure it out (i.e. Zygote hates it). I'm not sure why, and the error message is unfortunate. For this first pass, I think we should just do this, and then maybe bisect this choice and try and figure out why Zygote is not liking the other version in another issue/PR.

My understanding is that Lux handles this with less code. Can we do the same? I don't think a dedicated rule is required for foldl like theye have, however.

Yes, so if we try and use a similar implementation to the current Lux implementation it would look like the below.

function scan_full(func, init_carry, xs::AbstractVector{<:AbstractArray}) function __recurrence_op(::Nothing, input) carry, out = func(init_carry, input) return carry, [out] end # recurrence operation used in the fold. Takes the state of the # folde and the next input, returns the new state. function __recurrence_op((carry, outputs), input) carry, out = func(carry, input) return carry, vcat(outputs, [out]) end # Fold left to right. foldl(__recurrence_op, xs; init=nothing) end

Unfortunately, gradients won't compile afaict. I'm not sure why we don't get the same behavior as Lux.jl. Also preemptively Lux's foldl_init just calls an inline for foldl. Using foldl_init also fails in a similar way.

Here is the stack trace when running the tests. I have no idea what to takeaway from this error, and any help would be great.

Stack Trace

gradients-implicit: Error During Test at /Users/Matt/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:30 Got exception outside of a @test MethodError: Cannot `convert` an object of type Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Matrix{Float32}, Vector{Matrix{Float32}}},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Matrix{Float32}, Vector{Matrix{Float32}}},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2,1,Zygote.Context{true},Array{Matrix{Float32},1}}},Zygote.var"#2017#back#200"{typeof(identity)},Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}} to an object of type Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Nothing,SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Nothing,SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2,1,Zygote.Context{true},Array{Float32,2}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}},Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}},Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2017#back#200"{typeof(identity)}}}} Closest candidates are: convert(::Type{T}, ::T) where T @ Base Base.jl:64 Stacktrace: [1] cvt1 @ ./essentials.jl:418 [inlined] [2] ntuple @ ./ntuple.jl:49 [inlined] [3] convert(#unused#::Type{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}}}) @ Base ./essentials.jl:419 [4] setindex!(A::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}}}, i1::Int64) @ Base ./array.jl:969 [5] _accumulate1!(op::ChainRules.var"#1715#1717"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, v1::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dim::Int64) @ Base ./accumulate.jl:432 [6] _accumulate!(op::ChainRules.var"#1715#1717"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dims::Nothing, init::Some{Tuple{Nothing, Nothing}}) @ Base ./accumulate.jl:361 [7] accumulate!(op::Function, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Nothing, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}; dims::Nothing, kw::Base.Pairs{Symbol, Tuple{Nothing, Nothing}, Tuple{Symbol}, NamedTuple{(:init,), Tuple{Tuple{Nothing, Nothing}}}}) @ Base ./accumulate.jl:342 [8] accumulate! @ ./accumulate.jl:337 [inlined] [9] #accumulate#872 @ ./accumulate.jl:285 [inlined] [10] accumulate @ ./accumulate.jl:272 [inlined] [11] #rrule#1714 @ ~/.julia/packages/ChainRules/U21Ei/src/rulesets/Base/mapreduce.jl:440 [inlined] [12] rrule @ ~/.julia/packages/ChainRules/U21Ei/src/rulesets/Base/mapreduce.jl:430 [inlined] [13] chain_rrule_kw @ ~/.julia/packages/Zygote/JeHtr/src/compiler/chainrules.jl:235 [inlined] [14] macro expansion @ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined] [15] _pullback @ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined] [16] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:56 [inlined] [17] _pullback(::Zygote.Context{true}, ::typeof(Fluxperimental.scan_full), ::Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, ::Matrix{Float32}, ::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0 [18] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:70 [inlined] [19] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:142 [inlined] [20] _pullback(::Zygote.Context{true}, ::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, ::Matrix{Float32}, ::Array{Float32, 3}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0 [21] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:131 [inlined] [22] _pullback(ctx::Zygote.Context{true}, f::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, args::Array{Float32, 3}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0 [23] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:52 [inlined] [24] _pullback(::Zygote.Context{true}, ::var"#3#5"{Array{Float32, 3}, Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0 [25] pullback(f::Function, ps::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:384 [26] withgradient(f::Function, args::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:132 [27] macro expansion @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:51 [inlined] [28] macro expansion @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined] [29] macro expansion @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:31 [inlined] [30] macro expansion @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined] [31] top-level scope @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:4 [32] include(fname::String) @ Base.MainInclude ./client.jl:478 [33] macro expansion @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11 [inlined] [34] macro expansion @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined] [35] top-level scope @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11 [36] include(fname::String) @ Base.MainInclude ./client.jl:478 [37] top-level scope @ none:6 [38] eval @ ./boot.jl:370 [inlined] [39] exec_options(opts::Base.JLOptions) @ Base ./client.jl:280 [40] _start() @ Base ./client.jl:522

Aha I think I know what this is. Long story short, ChainRules tries to store an array of pullbacks, but since both method overloads have slightly different captured variables, they result in different pullbacks. Now I know why Lux has custom rrules for this: they erase the type of the pullback array to avoid the conversion error.

Does the following change work?

function scan_full(func, init_carry, xs::AbstractVector{<:AbstractArray}) function __recurrence_op(::Tuple{Nothing, Nothing}, input) carry, out = func(init_carry, input) return carry, [out] end # recurrence operation used in the fold. Takes the state of the function __recurrence_op((carry, outputs), input) ... end # Fold left to right. foldl(__recurrence_op, xs; init=(nothing, nothing)) end

What error do you get using the second one?

Seems similar to the previous error.

gradients-implicit: Error During Test at /Users/Matt/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:30 Got exception outside of a @test MethodError: Cannot `convert` an object of type Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Matrix{Float32},Vector{Matrix{Float32}}},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Matrix{Float32},Vector{Matrix{Float32}}},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2,1,Zygote.Context{true},Array{Matrix{Float32},1}}},Zygote.var"#2017#back#200"{typeof(identity)},Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}} to an object of type Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Nothing,Nothing},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Array{Float32{},2},Flux.RNNCell{typeof(identity){},Array{Float32{},2},Array{Float32{},2},Array{Float32{},1},Array{Float32{},2}}},Tuple{Nothing,Nothing},SubArray{Float32{},2,Array{Float32{},3},Tuple{Base.Slice{Base.OneTo{Int64{}}},Base.Slice{Base.OneTo{Int64{}}},Int64{}},true}},Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}},Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2,1,Zygote.Context{true},Array{Float32,2}}},Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}},Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}},Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}},Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}},Zygote.var"#2017#back#200"{typeof(identity)}}}} Closest candidates are: convert(::Type{T}, ::T) where T @ Base Base.jl:64 Stacktrace: [1] cvt1 @ ./essentials.jl:418 [inlined] [2] ntuple @ ./ntuple.jl:49 [inlined] [3] convert(#unused#::Type{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}}}) @ Base ./essentials.jl:419 [4] setindex!(A::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{Matrix{Float32}}}}}}}}}, i1::Int64) @ Base ./array.jl:969 [5] _accumulate1!(op::ChainRules.var"#1715#1717"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, v1::Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dim::Int64) @ Base ./accumulate.jl:432 [6] _accumulate!(op::ChainRules.var"#1715#1717"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dims::Nothing, init::Some{Tuple{Tuple{Nothing, Nothing}, Nothing}}) @ Base ./accumulate.jl:361 [7] accumulate!(op::Function, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{Matrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Nothing, Nothing}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:init_carry, Zygote.Context{true}, Fluxperimental.var"#__recurrence_op#35"{Matrix{Float32}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}; dims::Nothing, kw::Base.Pairs{Symbol, Tuple{Tuple{Nothing, Nothing}, Nothing}, Tuple{Symbol}, NamedTuple{(:init,), Tuple{Tuple{Tuple{Nothing, Nothing}, Nothing}}}}) @ Base ./accumulate.jl:342 [8] accumulate! @ ./accumulate.jl:337 [inlined] [9] #accumulate#872 @ ./accumulate.jl:285 [inlined] [10] accumulate @ ./accumulate.jl:272 [inlined] [11] #rrule#1714 @ ~/.julia/packages/ChainRules/U21Ei/src/rulesets/Base/mapreduce.jl:440 [inlined] [12] rrule @ ~/.julia/packages/ChainRules/U21Ei/src/rulesets/Base/mapreduce.jl:430 [inlined] [13] chain_rrule_kw @ ~/.julia/packages/Zygote/JeHtr/src/compiler/chainrules.jl:235 [inlined] [14] macro expansion @ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined] [15] _pullback @ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined] [16] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:50 [inlined] [17] _pullback(::Zygote.Context{true}, ::typeof(Fluxperimental.scan_full), ::Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, ::Matrix{Float32}, ::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0 [18] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:106 [inlined] [19] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:178 [inlined] [20] _pullback(::Zygote.Context{true}, ::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, ::Matrix{Float32}, ::Array{Float32, 3}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0 [21] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:167 [inlined] [22] _pullback(ctx::Zygote.Context{true}, f::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, args::Array{Float32, 3}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0 [23] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:52 [inlined] [24] _pullback(::Zygote.Context{true}, ::var"#3#5"{Array{Float32, 3}, Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0 [25] pullback(f::Function, ps::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:384 [26] withgradient(f::Function, args::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:132 [27] macro expansion @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:51 [inlined] [28] macro expansion @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined] [29] macro expansion @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:31 [inlined] [30] macro expansion @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined] [31] top-level scope @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:4 [32] include(fname::String) @ Base.MainInclude ./client.jl:478 [33] macro expansion @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11 [inlined] [34] macro expansion @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined] [35] top-level scope @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11 [36] include(fname::String) @ Base.MainInclude ./client.jl:478 [37] top-level scope @ none:6 [38] eval @ ./boot.jl:370 [inlined] [39] exec_options(opts::Base.JLOptions) @ Base ./client.jl:280 [40] _start() @ Base ./client.jl:522

/rulesets/Base/mapreduce.jl:440

This is https://github.com/JuliaDiff/ChainRules.jl/blob/main/src/rulesets/Base/mapreduce.jl#L440 which is part of the rule for foldl. Which I wrote and is a bit of a monster.

If it's easy to do so, trying with JuliaDiff/ChainRules.jl#569 might be worthwhile This re-writes the rule to solve various problems, esp. to do with init.

Using mapfoldl_impl would work well enough, here's a rough sketch:

function scan_full(func, init_carry, xs::AbstractVector{<:AbstractArray}) # Recurrence operation used in the fold. Takes the state of the # fold and the next input, returns the new state. function recurrence_op((carry, outputs), input) carry, out = func(carry, input) return carry, vcat(outputs, [out]) end # Fold left to right. return Base.mapfoldl_impl(identity, recurrence_op, (init_carry, empty(xs)), xs) end function rrule( config::RuleConfig{>:HasReverseMode}, ::typeof(Base.mapfoldl_impl), ::typeof(identity), op::G, init, x::Union{AbstractArray, Tuple}; ) where {G} hobbits = accumulate(x; init=(init, nothing)) do (a, _), b c, back = rrule_via_ad(config, op, a, b) end y = first(last(hobbits)) axe = axes(x) project = ProjectTo(x) function unfoldl(dy) trio = accumulate(Iterators.reverse(hobbits); init=(0, dy, 0)) do (_, dc, _), (_, back) ds, da, db = back(dc) end dop = sum(first, trio) dx = map(last, Iterators.reverse(trio)) d_init = trio[end][2] return (NoTangent(), NoTangent(), dop, d_init, project(reshape(dx, axe))) end return y, unfoldl end

I think the problem is that the rrule which takes an Array in that PR doesn't provide a gradient for the init value. I don't think it would be too hard to add (see simplified method above) but I may be missing something.

I'm getting a similar error still, and it is in the line hobbits = accumulate.... I'm not too sure how to debug this, as it is a convert issue when accumulating the gradient. When I get a bit of time I will try and debug a bit more, but I'm at a loss with the current error, so any help would be appreciated.

gradients-implicit: Error During Test at /Users/Matt/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:30 Got exception outside of a @test MethodError: Cannot `convert` an object of type Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}}}}} to an object of type Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}} Closest candidates are: convert(::Type{T}, ::T) where T @ Base Base.jl:64 Stacktrace: [1] cvt1 @ ./essentials.jl:418 [inlined] [2] ntuple @ ./ntuple.jl:49 [inlined] [3] convert(#unused#::Type{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}}}}}}) @ Base ./essentials.jl:419 [4] setindex!(A::Vector{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, x::Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{AbstractMatrix{Float32}}}}}}}}}, i1::Int64) @ Base ./array.jl:969 [5] _accumulate1!(op::Fluxperimental.var"#36#38"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, v1::Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Zygote.Pullback{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:func, Zygote.Context{true}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.ZBack{ChainRules.var"#vect_pullback#1369"{1, Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Int64}}, Zygote.Pullback{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wi, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.ZBack{Flux.var"#_size_check_pullback#204"{Tuple{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}, Pair{Int64, Int64}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Zygote.Pullback{Tuple{Type{Pair}, Int64, Int64}, Tuple{Zygote.var"#2214#back#309"{Zygote.Jnew{Pair{Int64, Int64}, Nothing, false}}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.ZBack{ChainRules.var"#fieldtype_pullback#421"}, Zygote.Pullback{Tuple{typeof(Core.convert), Type{Int64}, Int64}, Tuple{}}}}, Zygote.Pullback{Tuple{typeof(Flux.reshape_cell_output), Matrix{Float32}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{Zygote.var"#2173#back#289"{Zygote.var"#287#288"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.var"#2799#back#621"{Zygote.var"#615#619"{Matrix{Float32}, Tuple{Colon, Int64}}}}}, Zygote.Pullback{Tuple{typeof(lastindex), Tuple{Int64, Int64}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#747"}}}, Zygote.ZBack{ChainRules.var"#:_pullback#276"{Tuple{Int64, Int64}}}, Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.var"#2065#back#228"{Zygote.var"#222#226"{2, UnitRange{Int64}}}}}, Zygote.ZBack{Flux.var"#175#176"}, Zygote.var"#3898#back#1243"{Zygote.var"#1239#1242"}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#times_pullback#1481"{Matrix{Float32}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:σ, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, typeof(identity)}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:Wh, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Matrix{Float32}}}, Zygote.var"#2184#back#299"{Zygote.var"#back#298"{:b, Zygote.Context{true}, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, Vector{Float32}}}, Zygote.Pullback{Tuple{typeof(Base.Broadcast.materialize), Matrix{Float32}}, Tuple{}}, Zygote.var"#2017#back#200"{typeof(identity)}, Zygote.Pullback{Tuple{typeof(NNlib.fast_act), typeof(identity), SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, Tuple{}}, Zygote.var"#3754#back#1177"{Zygote.var"#1171#1175"{Tuple{Matrix{Float32}, Vector{Float32}}}}, Zygote.ZBack{ChainRules.var"#size_pullback#919"}}}, Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}, Zygote.var"#back#241"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 1, Zygote.Context{true}, Matrix{Float32}}}}, Zygote.ZBack{ChainRules.var"#vcat_pullback#1412"{Tuple{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{Any}, Tuple{Base.OneTo{Int64}}}}}, ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:elements, :axes), Tuple{Vector{ChainRulesCore.ProjectTo{AbstractArray, NamedTuple{(:element, :axes), Tuple{ChainRulesCore.ProjectTo{Float32, NamedTuple{(), Tuple{}}}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}}, Tuple{Base.OneTo{Int64}}}}}}, Tuple{Tuple{Int64}, Tuple{Int64}}, Val{1}}}, Zygote.var"#back#242"{Zygote.var"#2033#back#209"{Zygote.var"#back#207"{2, 2, Zygote.Context{true}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dim::Int64) @ Base ./accumulate.jl:432 [6] _accumulate!(op::Fluxperimental.var"#36#38"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}, dims::Nothing, init::Some{Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}}) @ Base ./accumulate.jl:361 [7] accumulate!(op::Function, B::Vector{Tuple{Tuple{Matrix{Float32}, Vector{AbstractMatrix{Float32}}}, Zygote.var"#ad_pullback#58"{Tuple{Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}}}, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}; dims::Nothing, kw::Base.Pairs{Symbol, Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}, Tuple{Symbol}, NamedTuple{(:init,), Tuple{Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}}}}) @ Base ./accumulate.jl:342 [8] accumulate(op::Function, A::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}; dims::Nothing, kw::Base.Pairs{Symbol, Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}, Tuple{Symbol}, NamedTuple{(:init,), Tuple{Tuple{Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, Nothing}}}}) @ Base ./accumulate.jl:285 [9] accumulate @ ./accumulate.jl:272 [inlined] [10] rrule(config::Zygote.ZygoteRuleConfig{Zygote.Context{true}}, ::typeof(Base.mapfoldl_impl), ::typeof(identity), op::Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, init::Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, x::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}) @ Fluxperimental ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:48 [11] chain_rrule(::Zygote.ZygoteRuleConfig{Zygote.Context{true}}, ::Function, ::Function, ::Vararg{Any}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/chainrules.jl:223 [12] macro expansion @ ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [inlined] [13] _pullback(::Zygote.Context{true}, ::typeof(Base.mapfoldl_impl), ::typeof(identity), ::Fluxperimental.var"#recurrence_op#35"{Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, ::Tuple{Matrix{Float32}, Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}}, ::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:101 [14] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:42 [inlined] [15] _pullback(::Zygote.Context{true}, ::typeof(Fluxperimental.scan_full), ::Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}, ::Matrix{Float32}, ::Vector{SubArray{Float32, 2, Array{Float32, 3}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, Int64}, true}}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0 [16] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:137 [inlined] [17] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:209 [inlined] [18] _pullback(::Zygote.Context{true}, ::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, ::Matrix{Float32}, ::Array{Float32, 3}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0 [19] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/src/new_recur.jl:198 [inlined] [20] _pullback(ctx::Zygote.Context{true}, f::Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}, args::Array{Float32, 3}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0 [21] _pullback @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:52 [inlined] [22] _pullback(::Zygote.Context{true}, ::var"#3#5"{Array{Float32, 3}, Fluxperimental.NewRecur{true, Flux.RNNCell{typeof(identity), Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Matrix{Float32}}}}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface2.jl:0 [23] pullback(f::Function, ps::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:384 [24] withgradient(f::Function, args::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}}) @ Zygote ~/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:132 [25] macro expansion @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:51 [inlined] [26] macro expansion @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined] [27] macro expansion @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:31 [inlined] [28] macro expansion @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined] [29] top-level scope @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/new_recur.jl:4 [30] include(fname::String) @ Base.MainInclude ./client.jl:478 [31] macro expansion @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11 [inlined] [32] macro expansion @ ~/.julia/juliaup/julia-1.9.1+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined] [33] top-level scope @ ~/Documents/Developer_Projects/Julia/Fluxperimental.jl/test/runtests.jl:11 [34] include(fname::String) @ Base.MainInclude ./client.jl:478 [35] top-level scope @ none:6 [36] eval @ ./boot.jl:370 [inlined] [37] exec_options(opts::Base.JLOptions) @ Base ./client.jl:280 [38] _start() @ Base ./client.jl:522

ToucheSir · 2023-07-08T18:21:31Z

src/new_recur.jl

+(l::NewRecur)(init_carry, x_mat::AbstractMatrix) = MethodError("Matrix is ambiguous with NewRecur")
+(l::NewRecur)(init_carry, x_mat::AbstractVector{T}) where {T<:Number} = MethodError("Vector is ambiguous with NewRecur")
+
+(l::NewRecur)(xs) = l(l.cell.state0, xs)


Suggested change

(l::NewRecur)(xs) = l(l.cell.state0, xs)

(l::NewRecur)(xs::AbstractArray) = l(l.cell.state0, xs)

For your consideration

Likely this would be a good idea. But I didn't restrict this in this initial pass.

src/new_recur.jl

ToucheSir · 2023-07-08T18:35:04Z

src/new_recur.jl

+function (l::NewRecur{true})(init_carry,
+                             xs,)
+
+  results = scan_full(l.cell, init_carry, xs)
+
+  h = results[2]
+  sze = size(h[1])
+  reshape(reduce(hcat, h), sze[1], sze[2], length(h))
+end


Suggested change

function (l::NewRecur{true})(init_carry,

xs,)

results = scan_full(l.cell, init_carry, xs)

h = results[2]

sze = size(h[1])

reshape(reduce(hcat, h), sze[1], sze[2], length(h))

end

function (l::NewRecur{true})(init_carry, xs)

results = scan_full(l.cell, init_carry, xs)

return results[1], stack(results[2])

end

Similar story here. stack vs reduce(hcat are more or less equally efficient so feel free to use either.

Changed this to stack like you suggested.

I didn't realize stack is not available in 1.6. This function was added to 1.9. Are we targeting only 1.9?

It's available via Compat.jl, which Flux already has as a transitive dep so there's zero additional import overhead.

ToucheSir · 2023-07-10T02:58:58Z

Here's a possibly crazy idea for handling carry values: can we split the NewRecur call function into one which takes a carry and returns a carry, and one which takes no carry and returns no carry? I suspect the latter will cover most use cases while avoiding any additional model complexity with container layers. Those who need the former are probably power users who would be more comfortable with advanced use of container layers and/or defining their own layers.

darsnack · 2023-07-10T12:26:55Z

Sorry that I haven't been following these discussions closely, but I'm catching up now pre-0.14.

Here's a possibly crazy idea for handling carry values: can we split the NewRecur call function into one which takes a carry and returns a carry, and one which takes no carry and returns no carry?

I prefer this option. To be honest, I don't really get the "optimization" of not returning everything. The performance difference is not doing a reshape, so it boils down to convenience within a Chain. Seems cleaner and more inline with other parts of Flux to split the struct, split the forward function, or wrap the struct vs. a flag type parameter.

mkschleg · 2023-07-10T16:28:40Z

I think that is a great compromise. We could also do a parameter option (like with SEQ) to handle this. But maybe that adds too much complexity to the interface?

I do think it might be difficult for new users to understand this particular overload though. I.e. changing the function arguments changes the return type, but that is generally the case in Julia so likely ok.

@darsnack All good! Just to be clear, you mean for the carry correct? We still will use a flag type parameter to determine whether to return the whole sequence or not (this is different than the carry question). Any reason not to do that (i.e. SEQ in the current impl).

@ToucheSir Thanks for the review! This definitely needed a pass from you, and a lot of these decisions came up in the other recur interface for the new chain. So documenting them here would be good. The delay actually worked for me due to job interviews 😆. Working on this today.

mkschleg · 2023-07-10T22:50:36Z

Ok. Took another pass and made almost all of your changes @ToucheSir. I didn't use the edits you made directly because there was some specific issues that needed to be resolved throughout to make the changes work.

I've not resolved the conversations for the specific changes that I either didn't do/which needed more info/conversation. And I reopened the stack due to it only being available on 1.9.

mkschleg · 2023-08-03T20:50:20Z

Ok. Cleaned up the code a bit and fixed the gradients. We are failing on 1.6, but because Flux now requires 1.9, I'm not sure that is an issue.

Unfortunately, hobbits seems to need to be a vector of Anys. I'm not sure why this is the case, and I haven't figured out a type signature for that vector which works.

ToucheSir · 2023-08-04T06:42:10Z

I think adding Compat as a dep and using stack from there instead of Base should be enough. Would be fine with bumping the min Julia compat as well though.

mkschleg · 2023-08-04T16:48:03Z

Likely the right solution would be to up the bounds for julia, but I just added Compat as a dependency. This project will probably need it anyway as time goes on.

ToucheSir

Do you mind removing all the extraneous newlines in {src,test}/new_recur.jl? Otherwise LGTM!

… newrecur

mkschleg · 2023-08-09T01:20:08Z

Done.

mkschleg added 2 commits June 25, 2023 17:58

First pass on NewRecur.

c007097

Remove comments.

c2a0ec9

ToucheSir reviewed Jun 26, 2023

View reviewed changes

src/new_recur.jl Outdated Show resolved Hide resolved

Updating interface, adding more tests and re-orging tests.

49c601c

Moving functionality to separate helper functions.

79e7261

mkschleg requested a review from ToucheSir June 27, 2023 19:28

ToucheSir reviewed Jul 8, 2023

View reviewed changes

Modified interface slightly according to comments.

b28bb57

mkschleg added 2 commits August 3, 2023 10:25

Fixed gradients using Lux's impl.

7b60350

Cleanup.

832f860

Fixed tests.

72a7fe1

Added Compat as a dependency.

b238091

Minor edits to whitespace, Temp Docs.

2ed6588

mkschleg requested a review from ToucheSir August 7, 2023 16:11

ToucheSir approved these changes Aug 8, 2023

View reviewed changes

mkschleg added 3 commits August 8, 2023 19:17

Remove extra newlines.

1761614

Merge branch 'newrecur' of github.com:mkschleg/Fluxperimental.jl into…

c4d92b1

… newrecur

Some more small modifications.

52f3b7f

ToucheSir merged commit 9dcae27 into FluxML:master Aug 9, 2023
3 checks passed

		(carry, out_) = func(init_carry, x_init)

		init = (typeof(out_)[out_], carry)

	(l::NewRecur)(xs) = l(l.cell.state0, xs)
	(l::NewRecur)(xs::AbstractArray) = l(l.cell.state0, xs)

NewRecur experimental interface #11

NewRecur experimental interface #11

Conversation

mkschleg commented Jun 26, 2023

PR Checklist

codecov-commenter commented Jun 26, 2023 • edited Loading

Codecov Report

mkschleg commented Jun 26, 2023

ToucheSir commented Jun 26, 2023 • edited Loading

mkschleg commented Jun 26, 2023 • edited Loading

mkschleg commented Jun 26, 2023

mkschleg commented Jun 26, 2023 • edited Loading

ToucheSir commented Jun 27, 2023

mkschleg commented Jun 27, 2023

mkschleg commented Jun 27, 2023

ToucheSir left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkschleg Jul 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkschleg Jul 10, 2023 • edited Loading

Choose a reason for hiding this comment

ToucheSir Jul 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ToucheSir commented Jul 10, 2023 • edited Loading

darsnack commented Jul 10, 2023

mkschleg commented Jul 10, 2023

mkschleg commented Jul 10, 2023

mkschleg commented Aug 3, 2023

ToucheSir commented Aug 4, 2023

mkschleg commented Aug 4, 2023

ToucheSir left a comment

Choose a reason for hiding this comment

mkschleg commented Aug 9, 2023

codecov-commenter commented Jun 26, 2023 •

edited

Loading

ToucheSir commented Jun 26, 2023 •

edited

Loading

mkschleg commented Jun 26, 2023 •

edited

Loading

mkschleg commented Jun 26, 2023 •

edited

Loading

mkschleg Jul 10, 2023 •

edited

Loading

mkschleg Jul 10, 2023 •

edited

Loading

ToucheSir Jul 11, 2023 •

edited

Loading

ToucheSir commented Jul 10, 2023 •

edited

Loading