Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Internal error from a BoundsError #960

Closed
EdAyers opened this issue Apr 28, 2021 · 11 comments · Fixed by JuliaDiff/ChainRules.jl#337
Closed

Internal error from a BoundsError #960

EdAyers opened this issue Apr 28, 2021 · 11 comments · Fixed by JuliaDiff/ChainRules.jl#337

Comments

@EdAyers
Copy link

EdAyers commented Apr 28, 2021

I was having a go at implementing an NN for representing images as a xy→rgb (eg see https://bmild.github.io/fourfeat/) in Julia using Flux. But I encountered an internal error in Julia and my understanding is that this is always a bug.

Versions:

  • Julia Version 1.6.0 (2021-03-24)
[deps]
CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba"
ColorTypes = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"
ColorVectorSpace = "c3611d14-8923-5661-9e6a-0046d554d3a4"
FileIO = "5789e2e9-d7fb-5bc7-8068-2c6fae9b9549"
Flux = "587475ba-b771-5e3f-ad9e-33799f191a9c"
ImageDraw = "4381153b-2b60-58ae-a1ba-fd683676385f"
ImageShow = "4e3cecfd-b093-5904-9786-8bbb286a6a31"
Pluto = "c3e4b0f8-55cb-11ea-2926-15256bba5781"
Revise = "295af30f-e4ad-537b-8983-00126c2a3abe"

Reproduction code:

using ImageShow, ColorVectorSpace, ColorTypes, Flux, StaticArrays

(H,W) = (100, 200)
width = 256

col(x) = ColorTypes.RGB(x...)

model = (Chain(
		Dense(2, width, relu),
		Dense(width, width, relu),
		Dense(width, width, relu),
		Dense(width, 3, sigmoid),
	))

ι = ([SA[i, j] for i=range(0.0, 1.0, length=H), j=range(0.0, 1.0, length=W)])

imtrue(x,y) = SA[cos(x*2) ^ 2, 0, cos(x + y) ^ 2]

im3 = [imtrue(x,y) for x=range(0.0, 1.0, length=H), y=range(0.0, 1.0, length=W)]

data = [(ι, im3)]

function loss(xy, rgb)
	sum(norm.(model.(xy) .- rgb))
end

println(loss(ι, im3))

opt = Descent(0.01)
Flux.train!(loss, Flux.params(model), data, opt)

@show col.(model.(ι))

Expected output: it trains the network. I am not using cu for CUDA explicitly so my understanding is that all of the above happens on the CPU.

Output:

Internal error: encountered unexpected error in runtime:
BoundsError(a=Array{Any, (2,)}[
  Core.Compiler.VarState(typ=Zygote.Pullback{Tuple{typeof(StaticArrays._norm), StaticArrays.Size{(3,)}, StaticArrays.SArray{Tuple{3}, Float64, 1, 3}}, Any}, undef=false),
  Core.Compiler.VarState(typ=Float64, undef=false)], i=(3,))
jl_bounds_error_ints at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7fdfea5605bd)
unknown function (ip: 0x7fdfea55d038)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea546b0c)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55cdb5)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea546b0c)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55cdb5)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea55163b)
unknown function (ip: 0x7fdfea51880f)
unknown function (ip: 0x7fdfea27c3f0)
unknown function (ip: 0x7fdfea549347)
unknown function (ip: 0x7fdfea546a31)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55cdb5)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea549bf6)
unknown function (ip: 0x7fdfea546a31)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55cdb5)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea549bf6)
unknown function (ip: 0x7fdfea546a31)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55de61)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea55163b)
unknown function (ip: 0x7fdfea51880f)
unknown function (ip: 0x7fdfea5189b2)
unknown function (ip: 0x7fdfea540127)
unknown function (ip: 0x7fdfea549827)
unknown function (ip: 0x7fdfea546a31)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55de61)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea549bf6)
unknown function (ip: 0x7fdfea546a31)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55cdb5)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea549bf6)
unknown function (ip: 0x7fdfea546a31)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55de61)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea546b0c)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55cdb5)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea546b0c)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55de61)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea546b0c)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55cdb5)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea546b0c)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55cdb5)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea546b0c)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55cdb5)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea546b0c)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55de61)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea546b0c)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55cdb5)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea546b0c)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55cdb5)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea51effd)
unknown function (ip: 0x7fdfea51df88)
unknown function (ip: 0x7fdfea51b795)
unknown function (ip: 0x7fdfea546b0c)
unknown function (ip: 0x7fdfea54634f)
unknown function (ip: 0x7fdfea544795)
unknown function (ip: 0x7fdfea55cdb5)
unknown function (ip: 0x7fdfea559422)
unknown function (ip: 0x7fdfea558603)
unknown function (ip: 0x7fdfea55709a)
unknown function (ip: 0x7fdfea513c35)
unknown function (ip: 0x7fdfea512cfb)
unknown function (ip: 0x7fdfea289594)
unknown function (ip: 0x7fdfea2895cf)
unknown function (ip: 0x7fdffb5bcf8d)
unknown function (ip: 0x7fdffb642b70)
unknown function (ip: 0x7fdffb5bb8df)
jl_apply_generic at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
gradient at /home/edward/.julia/packages/Zygote/6HN9x/src/compiler/interface.jl:59
unknown function (ip: 0x7fdf5548d2fe)
macro expansion at /home/edward/.julia/packages/Flux/6BByF/src/optimise/train.jl:101 [inlined]
macro expansion at /home/edward/.julia/packages/Juno/n6wyj/src/progress.jl:134 [inlined]
#train!#36 at /home/edward/.julia/packages/Flux/6BByF/src/optimise/train.jl:99
train! at /home/edward/.julia/packages/Flux/6BByF/src/optimise/train.jl:97
unknown function (ip: 0x7fdf5548c381)
unknown function (ip: 0x7fdffb5d52c5)
unknown function (ip: 0x7fdffb5d4e85)
unknown function (ip: 0x7fdffb5d5a80)
unknown function (ip: 0x7fdffb5d6580)
top-level scope at /home/edward/Nerf.jl/train.jl:28
unknown function (ip: 0x7fdffb5f1777)
unknown function (ip: 0x7fdffb5f2407)
jl_toplevel_eval_in at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7fdfea583877)
unknown function (ip: 0x7fdfea48090b)
unknown function (ip: 0x7fdfea480372)
unknown function (ip: 0x7fdfea21bdc9)
unknown function (ip: 0x7fdfea2207dc)
unknown function (ip: 0x7fdfea220945)
unknown function (ip: 0x7fdffb6148de)
repl_entrypoint at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
main at julia (unknown line)
__libc_start_main at /usr/bin/../lib/libc.so.6 (unknown line)
_start at julia (unknown line)
ERROR: LoadError: BoundsError: attempt to access 2-element Vector{Any} at index [3]
Stacktrace:
  [1] return_type(interp::Core.Compiler.NativeInterpreter, f::Any, t::Any)
    @ Core.Compiler ./compiler/typeinfer.jl:963
  [2] return_type(f::Any, t::Any, world::UInt64)
    @ Core.Compiler ./compiler/typeinfer.jl:950
  [3] return_type(f::Any, t::Any)
    @ Core.Compiler ./compiler/typeinfer.jl:947
  [4] collect(itr::Base.Generator)
    @ Base ./array.jl:652
  [5] map(::Function, ::Matrix{typeof(∂(norm))}, ::FillArrays.Fill{Float64, 2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}})
    @ Base ./abstractarray.jl:2383
  [6] (::Zygote.var"#1077#1084")(ȳ::Any)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/lib/broadcast.jl:164
  [7] (::Zygote.var"#3902#back#1088")(Δ::Any)
    @ Zygote ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
  [8] (::Zygote.var"#180#181")(Δ::Any)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/lib/lib.jl:194
  [9] (::Zygote.var"#1689#back#182")(Δ::Any)
    @ Zygote ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [10] Pullback
    @ ./broadcast.jl:1309 [inlined]
 [11] (::typeof(∂(broadcasted)))(Δ::FillArrays.Fill{Float64, 2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}})
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [12] Pullback
    @ ~/Nerf.jl/train.jl:24 [inlined]
 [13] (::typeof(∂(loss)))(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [14] (::Zygote.var"#180#181")(Δ::Any)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/lib/lib.jl:194
 [15] (::Zygote.var"#1689#back#182")(Δ::Any)
    @ Zygote ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [16] Pullback
    @ ~/.julia/packages/Flux/6BByF/src/optimise/train.jl:102 [inlined]
 [17] (::typeof(∂(λ)))(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [18] (::Zygote.var"#69#70")(Δ::Any)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface.jl:252
 [19] gradient(f::Function, args::Zygote.Params)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface.jl:59
 [20] macro expansion
    @ ~/.julia/packages/Flux/6BByF/src/optimise/train.jl:101 [inlined]
 [21] macro expansion
    @ ~/.julia/packages/Juno/n6wyj/src/progress.jl:134 [inlined]
 [22] train!(loss::Function, ps::Zygote.Params, data::Vector{Tuple{Matrix{SVector{2, Float64}}, Matrix{SVector{3, Float64}}}}, opt::Descent; cb::Flux.Optimise.var"#40#46")
    @ Flux.Optimise ~/.julia/packages/Flux/6BByF/src/optimise/train.jl:99
 [23] train!(loss::Function, ps::Zygote.Params, data::Vector{Tuple{Matrix{SVector{2, Float64}}, Matrix{SVector{3, Float64}}}}, opt::Descent)
    @ Flux.Optimise ~/.julia/packages/Flux/6BByF/src/optimise/train.jl:97
 [24] top-level scope
    @ ~/Nerf.jl/train.jl:28
in expression starting at /home/edward/Nerf.jl/train.jl:28
@ericphanson
Copy link

Just to say I could reproduce this on Julia 1.6.1 on MacOS, though with a slightly different looking stacktrace:

julia> Flux.train!(loss, Flux.params(model), data, opt)
Internal error: encountered unexpected error in runtime:
BoundsError(a=Array{Any, (2,)}[
  Core.Compiler.VarState(typ=Zygote.Pullback{Tuple{typeof(StaticArrays._norm), StaticArrays.Size{(3,)}, StaticArrays.SArray{Tuple{3}, Float64, 1, 3}}, Any}, undef=false),
  Core.Compiler.VarState(typ=Float64, undef=false)], i=(3,))
jl_bounds_error_ints at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
setindex! at ./essentials.jl:458 [inlined]
stupdate! at ./compiler/typelattice.jl:279
typeinf_local at ./compiler/abstractinterpretation.jl:1493
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call at ./compiler/abstractinterpretation.jl:1054
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call at ./compiler/abstractinterpretation.jl:1054
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call_known at ./compiler/abstractinterpretation.jl:1033
abstract_call at ./compiler/abstractinterpretation.jl:1056
abstract_apply at ./compiler/abstractinterpretation.jl:738
abstract_call_known at ./compiler/abstractinterpretation.jl:952
abstract_call at ./compiler/abstractinterpretation.jl:1056
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call_known at ./compiler/abstractinterpretation.jl:1033
abstract_call at ./compiler/abstractinterpretation.jl:1056
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call_known at ./compiler/abstractinterpretation.jl:1033
abstract_call at ./compiler/abstractinterpretation.jl:1056
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1447
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call_known at ./compiler/abstractinterpretation.jl:1033
abstract_call at ./compiler/abstractinterpretation.jl:1056
jfptr_abstract_call_11621 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
return_type_tfunc at ./compiler/tfuncs.jl:1584
abstract_call_known at ./compiler/abstractinterpretation.jl:982
abstract_call at ./compiler/abstractinterpretation.jl:1056
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1447
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call_known at ./compiler/abstractinterpretation.jl:1033
abstract_call at ./compiler/abstractinterpretation.jl:1056
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call_known at ./compiler/abstractinterpretation.jl:1033
abstract_call at ./compiler/abstractinterpretation.jl:1056
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1447
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call at ./compiler/abstractinterpretation.jl:1054
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call at ./compiler/abstractinterpretation.jl:1054
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1447
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call at ./compiler/abstractinterpretation.jl:1054
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call at ./compiler/abstractinterpretation.jl:1054
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call at ./compiler/abstractinterpretation.jl:1054
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call at ./compiler/abstractinterpretation.jl:1054
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1447
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call at ./compiler/abstractinterpretation.jl:1054
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call at ./compiler/abstractinterpretation.jl:1054
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call at ./compiler/abstractinterpretation.jl:1054
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_ext at ./compiler/typeinfer.jl:892
typeinf_ext_toplevel at ./compiler/typeinfer.jl:925
typeinf_ext_toplevel at ./compiler/typeinfer.jl:921
jfptr_typeinf_ext_toplevel_12207 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_type_infer at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_generate_fptr at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_compile_method_internal at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
gradient at /Users/eph/.julia/packages/Zygote/6HN9x/src/compiler/interface.jl:59
unknown function (ip: 0x16748c594)
macro expansion at /Users/eph/.julia/packages/Flux/6BByF/src/optimise/train.jl:101 [inlined]
macro expansion at /Users/eph/.julia/packages/Juno/n6wyj/src/progress.jl:134 [inlined]
#train!#36 at /Users/eph/.julia/packages/Flux/6BByF/src/optimise/train.jl:99
train! at /Users/eph/.julia/packages/Flux/6BByF/src/optimise/train.jl:97
unknown function (ip: 0x16748b50e)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
do_call at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
eval_body at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_interpret_toplevel_thunk at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
top-level scope at REPL[24]:1
jl_toplevel_eval_flex at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
eval_body at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
eval_body at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_interpret_toplevel_thunk at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
top-level scope at /Users/eph/.julia/packages/CUDA/Px7QU/src/initialization.jl:81
jl_toplevel_eval_flex at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_toplevel_eval_in at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
eval at ./boot.jl:360 [inlined]
eval_user_input at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:139
repl_backend_loop at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:200
start_repl_backend at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:185
#run_repl#42 at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:317
run_repl at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:305
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
#874 at ./client.jl:387
jfptr_YY.874_25971 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_f__call_latest at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
#invokelatest#2 at ./essentials.jl:708 [inlined]
invokelatest at ./essentials.jl:706 [inlined]
run_main_repl at ./client.jl:372
exec_options at ./client.jl:302
_start at ./client.jl:485
jfptr__start_32022 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
true_main at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
repl_entrypoint at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
ERROR: BoundsError: attempt to access 2-element Vector{Any} at index [3]
Stacktrace:
  [1] setindex!
    @ ./essentials.jl:458 [inlined]
  [2] stupdate!(state::Nothing, changes::Core.Compiler.StateUpdate)
    @ Core.Compiler ./compiler/typelattice.jl:279
  [3] typeinf_local(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1493
  [4] typeinf_nocycle(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1520
  [5] _typeinf(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/typeinfer.jl:214
  [6] typeinf(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/typeinfer.jl:209
  [7] typeinf_edge(interp::Core.Compiler.NativeInterpreter, method::Method, atypes::Any, sparams::Core.SimpleVector, caller::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/typeinfer.jl:806
  [8] abstract_call_method(interp::Core.Compiler.NativeInterpreter, method::Method, sig::Any, sparams::Core.SimpleVector, hardlimit::Bool, sv::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:490
  [9] abstract_call_gf_by_type(interp::Core.Compiler.NativeInterpreter, f::Any, argtypes::Vector{Any}, atype::Any, sv::Core.Compiler.InferenceState, max_methods::Int64)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:143
 [10] abstract_call(interp::Core.Compiler.NativeInterpreter, fargs::Vector{Any}, argtypes::Vector{Any}, sv::Core.Compiler.InferenceState, max_methods::Int64)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1054
 [11] abstract_call(interp::Core.Compiler.NativeInterpreter, fargs::Vector{Any}, argtypes::Vector{Any}, sv::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1040
 [12] abstract_eval_statement(interp::Core.Compiler.NativeInterpreter, e::Any, vtypes::Vector{Any}, sv::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1167
 [13] typeinf_local(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1462
--- the last 10 lines are repeated 1 more time ---
 [24] typeinf_nocycle(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1520
 [25] _typeinf(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/typeinfer.jl:214
 [26] typeinf(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/typeinfer.jl:209
 [27] typeinf_edge(interp::Core.Compiler.NativeInterpreter, method::Method, atypes::Any, sparams::Core.SimpleVector, caller::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/typeinfer.jl:806
 [28] abstract_call_method(interp::Core.Compiler.NativeInterpreter, method::Method, sig::Any, sparams::Core.SimpleVector, hardlimit::Bool, sv::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:490
 [29] abstract_call_gf_by_type(interp::Core.Compiler.NativeInterpreter, f::Any, argtypes::Vector{Any}, atype::Any, sv::Core.Compiler.InferenceState, max_methods::Int64)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:143
 [30] abstract_call_known(interp::Core.Compiler.NativeInterpreter, f::Any, fargs::Nothing, argtypes::Vector{Any}, sv::Core.Compiler.InferenceState, max_methods::Int64)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1033
 [31] abstract_call(interp::Core.Compiler.NativeInterpreter, fargs::Nothing, argtypes::Vector{Any}, sv::Core.Compiler.InferenceState, max_methods::Int64)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1056
 [32] abstract_apply(interp::Core.Compiler.NativeInterpreter, itft::Any, aft::Any, aargtypes::Vector{Any}, sv::Core.Compiler.InferenceState, max_methods::Int64)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:738
 [33] abstract_call_known(interp::Core.Compiler.NativeInterpreter, f::Any, fargs::Vector{Any}, argtypes::Vector{Any}, sv::Core.Compiler.InferenceState, max_methods::Int64)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:952
 [34] abstract_call(interp::Core.Compiler.NativeInterpreter, fargs::Vector{Any}, argtypes::Vector{Any}, sv::Core.Compiler.InferenceState, max_methods::Int64)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1056
 [35] abstract_call(interp::Core.Compiler.NativeInterpreter, fargs::Vector{Any}, argtypes::Vector{Any}, sv::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1040
 [36] abstract_eval_statement(interp::Core.Compiler.NativeInterpreter, e::Any, vtypes::Vector{Any}, sv::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1167
 [37] typeinf_local(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1462
 [38] typeinf_nocycle(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1520
 [39] _typeinf(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/typeinfer.jl:214
 [40] typeinf(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/typeinfer.jl:209
 [41] typeinf_edge(interp::Core.Compiler.NativeInterpreter, method::Method, atypes::Any, sparams::Core.SimpleVector, caller::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/typeinfer.jl:806
 [42] abstract_call_method(interp::Core.Compiler.NativeInterpreter, method::Method, sig::Any, sparams::Core.SimpleVector, hardlimit::Bool, sv::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:490
 [43] abstract_call_gf_by_type(interp::Core.Compiler.NativeInterpreter, f::Any, argtypes::Vector{Any}, atype::Any, sv::Core.Compiler.InferenceState, max_methods::Int64)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:143
 [44] abstract_call_known(interp::Core.Compiler.NativeInterpreter, f::Any, fargs::Vector{Any}, argtypes::Vector{Any}, sv::Core.Compiler.InferenceState, max_methods::Int64)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1033
--- the last 11 lines are repeated 1 more time ---
 [56] abstract_call(interp::Core.Compiler.NativeInterpreter, fargs::Vector{Any}, argtypes::Vector{Any}, sv::Core.Compiler.InferenceState, max_methods::Int64)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1056
 [57] abstract_call(interp::Core.Compiler.NativeInterpreter, fargs::Vector{Any}, argtypes::Vector{Any}, sv::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1040
 [58] abstract_eval_statement(interp::Core.Compiler.NativeInterpreter, e::Any, vtypes::Vector{Any}, sv::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1167
 [59] typeinf_local(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1447
 [60] typeinf_nocycle(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/abstractinterpretation.jl:1520
 [61] _typeinf(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/typeinfer.jl:214
 [62] typeinf(interp::Core.Compiler.NativeInterpreter, frame::Core.Compiler.InferenceState)
    @ Core.Compiler ./compiler/typeinfer.jl:209
 [63] typeinf
    @ ./compiler/typeinfer.jl:8 [inlined]
 [64] typeinf_type(interp::Core.Compiler.NativeInterpreter, method::Method, atypes::Any, sparams::Core.SimpleVector)
    @ Core.Compiler ./compiler/typeinfer.jl:914
 [65] return_type(interp::Core.Compiler.NativeInterpreter, f::Any, t::Any)
    @ Core.Compiler ./compiler/typeinfer.jl:963
 [66] return_type(f::Any, t::Any, world::UInt64)
    @ Core.Compiler ./compiler/typeinfer.jl:950
 [67] return_type(f::Any, t::Any)
    @ Core.Compiler ./compiler/typeinfer.jl:947
 [68] collect(itr::Base.Generator)
    @ Base ./array.jl:652
 [69] map(::Function, ::Matrix{typeof(∂(norm))}, ::FillArrays.Fill{Float64, 2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}})
    @ Base ./abstractarray.jl:2383
 [70] (::Zygote.var"#1077#1084")(ȳ::Any)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/lib/broadcast.jl:164
 [71] (::Zygote.var"#3902#back#1088")(Δ::Any)
    @ Zygote ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [72] (::Zygote.var"#180#181")(Δ::Any)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/lib/lib.jl:194
 [73] (::Zygote.var"#1689#back#182")(Δ::Any)
    @ Zygote ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [74] Pullback
    @ ./broadcast.jl:1309 [inlined]
 [75] (::typeof(∂(broadcasted)))(Δ::FillArrays.Fill{Float64, 2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}})
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [76] Pullback
    @ ./REPL[22]:2 [inlined]
 [77] (::typeof(∂(loss)))(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [78] (::Zygote.var"#180#181")(Δ::Any)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/lib/lib.jl:194
 [79] (::Zygote.var"#1689#back#182")(Δ::Any)
    @ Zygote ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [80] Pullback
    @ ~/.julia/packages/Flux/6BByF/src/optimise/train.jl:102 [inlined]
 [81] (::typeof(∂(λ)))(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface2.jl:0
 [82] (::Zygote.var"#69#70")(Δ::Any)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface.jl:252
 [83] gradient(f::Function, args::Zygote.Params)
    @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface.jl:59
 [84] macro expansion
    @ ~/.julia/packages/Flux/6BByF/src/optimise/train.jl:101 [inlined]
 [85] macro expansion
    @ ~/.julia/packages/Juno/n6wyj/src/progress.jl:134 [inlined]
 [86] train!(loss::Function, ps::Zygote.Params, data::Vector{Tuple{Matrix{SVector{2, Float64}}, Matrix{SVector{3, Float64}}}}, opt::Descent; cb::Flux.Optimise.var"#40#46")
    @ Flux.Optimise ~/.julia/packages/Flux/6BByF/src/optimise/train.jl:99
 [87] train!(loss::Function, ps::Zygote.Params, data::Vector{Tuple{Matrix{SVector{2, Float64}}, Matrix{SVector{3, Float64}}}}, opt::Descent)
    @ Flux.Optimise ~/.julia/packages/Flux/6BByF/src/optimise/train.jl:97
 [88] top-level scope
    @ REPL[24]:1

@DhairyaLGandhi
Copy link
Member

Interesting indeed. Could you try without static arrays?

@EdAyers
Copy link
Author

EdAyers commented Apr 28, 2021

Yeah it works without the static arrays

@DhairyaLGandhi
Copy link
Member

A quick follow up, does it also work with cuda (without involving static arrays)?

@EdAyers
Copy link
Author

EdAyers commented Apr 28, 2021

I couldn't get it to work with CUDA without static arrays because it complained about not being a bits type or something. I am a complete beginner at Julia so I don't really know.

@DhairyaLGandhi
Copy link
Member

Ah yeah, CUDA doesn't work with non bitstype objects indeed.

@ericphanson
Copy link

As a workaround, it seems to not error if I do

square(x) = x^2
my_norm(vec) = sqrt(sum(square, vec))
function loss(xy, rgb)
	sum(my_norm.(model.(xy) .- rgb))
end

as suggested by @mcabbott on Slack.

@mcabbott mcabbott mentioned this issue Apr 29, 2021
Merged
@simeonschaub
Copy link
Member

simeonschaub commented Apr 29, 2021 •
edited
Loading

MWE is

Zygote._pullback(Zygote.Context(), StaticArrays._norm, Size(3), SA[1., 2, 3])[2](1.)

Looks like Zygote.Pullback is not handling generated functions correctly somewhere.

@mcabbott
Copy link
Member

Or less ambitiously, gradient(norm, SA[1, 2, 3])[1] ≈ gradient(norm, 1:3.0)[1].

@simeonschaub
Copy link
Member

We should definitely be able to handle recursing into _norm as well though, since the actual code generated is not hard to differentiate.

@EdAyers
Copy link
Author

EdAyers commented Apr 29, 2021 •
edited
Loading

A link to this issue #860 arose in the slack discussion

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Improvements to rules for norm mcabbott/ChainRules.jl
5 participants
@EdAyers @simeonschaub @ericphanson @mcabbott @DhairyaLGandhi