Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inference performance regression vs 1.2 (NeuralNetDiffEq) #33337

Closed
KristofferC opened this issue Sep 20, 2019 · 3 comments · Fixed by #33353
Closed

Inference performance regression vs 1.2 (NeuralNetDiffEq) #33337

KristofferC opened this issue Sep 20, 2019 · 3 comments · Fixed by #33353
Labels
compiler:latency Compiler latency regression Regression in behavior compared to a previous version
Milestone
1.3

Comments

@KristofferC
Copy link
Member

Using NeuralNetDiffEq with the code

using NeuralNetDiffEq, Flux
linear = (u,p,t) -> @. t^3 + 2*t + (t^2)*((1+3*(t^2))/(1+t+(t^3))) - u*(t + ((1+3*(t^2))/(1+t+t^3)))
linear_analytic = (u0,p,t) -> [exp(-(t^2)/2)/(1+t+t^3) + t^2]
prob = ODEProblem(ODEFunction(linear,analytic=linear_analytic),[1f0],(0.0f0,1.0f0))
chain = Flux.Chain(Dense(1,5,σ),Dense(5,1))
opt = Flux.ADAM(0.1, (0.9, 0.95))
sol  = solve(prob,NeuralNetDiffEq.NNODE(chain,opt),verbose = true, dt=1/5f0)

it finishes in about 8 seconds on 1.2 while on 1.3 it never seems to start. Interrupting it gives the following stacktrace but not sure how useful that is:

julia> @time sol  = solve(prob,NeuralNetDiffEq.NNODE(chain,opt),verbose = true, dt=1/5f0)
j^C^C^C^C^CWARNING: Force throwing a SIGINT
Internal error: encountered unexpected error in runtime:
InterruptException()
^C^C^C^Chas_free_typevars at /Users/julia/buildbot/worker/package_macos64/build/src/jltypes.c:162
has_free_typevars at /Users/julia/buildbot/worker/package_macos64/build/src/jltypes.c:159
has_free_typevars at /Users/julia/buildbot/worker/package_macos64/build/src/jltypes.c:0
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1054
with_tvar at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:701
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:0
subtype_tuple at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1145 [inlined]
subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1298
forall_exists_equal at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1350
subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1312
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1063
subtype_tuple at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1145 [inlined]
subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1298
with_tvar at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:701
subtype_unionall at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:836
with_tvar at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:732
subtype_unionall at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:836
exists_subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1374 [inlined]
forall_exists_subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1402
jl_subtype_env at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1767
ml_matches_visitor at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2631
jl_typemap_intersection_node_visitor at /Users/julia/buildbot/worker/package_macos64/build/src/./julia.h:0
jl_typemap_intersection_visitor at /Users/julia/buildbot/worker/package_macos64/build/src/typemap.c:572
jl_typemap_intersection_visitor at /Users/julia/buildbot/worker/package_macos64/build/src/typemap.c:563
ml_matches at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2701
jl_matching_methods at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:1803
abstract_call_gf_by_type at ./reflection.jl:821
abstract_call at ./compiler/abstractinterpretation.jl:846
abstract_call at ./compiler/abstractinterpretation.jl:636
abstract_eval_call at ./compiler/abstractinterpretation.jl:875
abstract_eval at ./compiler/abstractinterpretation.jl:945
typeinf_local at ./compiler/abstractinterpretation.jl:1204
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1260
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:488
abstract_call_method at ./compiler/abstractinterpretation.jl:396
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:93
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:21
abstract_eval_call at ./compiler/abstractinterpretation.jl:873
abstract_eval at ./compiler/abstractinterpretation.jl:945
typeinf_local at ./compiler/abstractinterpretation.jl:1204
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1260
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:488
abstract_call_method at ./compiler/abstractinterpretation.jl:396
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:93
abstract_call at ./compiler/abstractinterpretation.jl:846
abstract_call at ./compiler/abstractinterpretation.jl:636
abstract_eval_call at ./compiler/abstractinterpretation.jl:875
abstract_eval at ./compiler/abstractinterpretation.jl:945
typeinf_local at ./compiler/abstractinterpretation.jl:1204
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1260
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:488
abstract_call_method at ./compiler/abstractinterpretation.jl:396
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:93
abstract_call at ./compiler/abstractinterpretation.jl:846
jfptr_abstract_call_606.clone_1 at /Applications/Julia-1.3.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
abstract_apply at ./compiler/abstractinterpretation.jl:581
abstract_call at ./compiler/abstractinterpretation.jl:637
abstract_call at ./compiler/abstractinterpretation.jl:636
abstract_eval_call at ./compiler/abstractinterpretation.jl:875
abstract_eval at ./compiler/abstractinterpretation.jl:945
typeinf_local at ./compiler/abstractinterpretation.jl:1204
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1260
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:488
abstract_call_method at ./compiler/abstractinterpretation.jl:396
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:93
abstract_call at ./compiler/abstractinterpretation.jl:846
abstract_call at ./compiler/abstractinterpretation.jl:636
abstract_eval_call at ./compiler/abstractinterpretation.jl:875
abstract_eval at ./compiler/abstractinterpretation.jl:945
typeinf_local at ./compiler/abstractinterpretation.jl:1204
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1260
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:488
abstract_call_method at ./compiler/abstractinterpretation.jl:396
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:93
abstract_call at ./compiler/abstractinterpretation.jl:846
abstract_call at ./compiler/abstractinterpretation.jl:636
abstract_eval_call at ./compiler/abstractinterpretation.jl:875
abstract_eval at ./compiler/abstractinterpretation.jl:945
typeinf_local at ./compiler/abstractinterpretation.jl:1204
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1260
typeinf at ./compiler/typeinfer.jl:12
typeinf_ext at ./compiler/typeinfer.jl:574
typeinf_ext at ./compiler/typeinfer.jl:605
jfptr_typeinf_ext_1.clone_1 at /Applications/Julia-1.3.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
_jl_invoke at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2130 [inlined]
jl_apply_generic at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2300 [inlined]
jl_apply at /Users/julia/buildbot/worker/package_macos64/build/src/./julia.h:1631 [inlined]
jl_type_infer at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:209
jl_compile_method_internal at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:1877
_jl_invoke at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2135 [inlined]
jl_apply_generic at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2300
ODEFunction at /Users/kristoffer/.julia/packages/DiffEqBase/8uyX3/src/diffeqfunction.jl:230
#9 at ./none:0
mapfoldl_impl at ./generator.jl:47
#mapfoldl#186 at ./reduce.jl:72 [inlined]
mapfoldl at ./reduce.jl:72 [inlined]
#mapreduce#194 at ./reduce.jl:200 [inlined]
mapreduce at ./reduce.jl:200 [inlined]
sum at ./reduce.jl:395 [inlined]
#8 at /Users/kristoffer/Regressions13/dev/NeuralNetDiffEq/src/ode_solve.jl:47 [inlined]
#15 at /Users/kristoffer/.julia/packages/Flux/dkJUV/src/optimise/train.jl:72 [inlined]
gradient_ at /Users/kristoffer/.julia/packages/Tracker/SAr25/src/back.jl:97
#gradient#24 at /Users/kristoffer/.julia/packages/Tracker/SAr25/src/back.jl:164
unknown function (ip: 0x1215ac2ad)
gradient at /Users/kristoffer/.julia/packages/Tracker/SAr25/src/back.jl:164 [inlined]
macro expansion at /Users/kristoffer/.julia/packages/Flux/dkJUV/src/optimise/train.jl:71 [inlined]
macro expansion at /Users/kristoffer/.julia/packages/Juno/oLB1d/src/progress.jl:134 [inlined]
#train!#12 at /Users/kristoffer/.julia/packages/Flux/dkJUV/src/optimise/train.jl:69
#train! at ./none:0
unknown function (ip: 0x1215a0ae4)
#solve#2 at /Users/kristoffer/Regressions13/dev/NeuralNetDiffEq/src/ode_solve.jl:55
#solve at ./none:0
@KristofferC KristofferC added regression Regression in behavior compared to a previous version compiler:latency Compiler latency labels Sep 20, 2019
@KristofferC KristofferC added this to the 1.3 milestone Sep 20, 2019
@KristofferC KristofferC changed the title Inference performance regression vs 1.2 (NeuralNedDiffEq) Inference performance regression vs 1.2 (NeuralNetDiffEq) Sep 20, 2019
@KristofferC KristofferC mentioned this issue Sep 20, 2019
Merged
18 tasks
@JeffBezanson
Copy link
Member

Looking into this.

@JeffBezanson
Copy link
Member

Bisect points to

commit 06b4af4d57182de8d5c3ea21525924cc62c3bfaa
Author: Jameson Nash <vtjnash@gmail.com>
Date:   Fri Jun 28 12:20:33 2019 -0400

    dispatch: tolerate un-sorted tables and support non-transitive specificity

@JeffBezanson
Copy link
Member

We are in a subtyping problem that takes ~forever inside

julia/src/gf.c

Line 2631 in bc36752

if (jl_subtype(closure->match.ti, sig2)) {
.

It's something like

julia> B = Tuple{typeof(Base.convert), Type{Tuple{Vararg{V, N} where N}}, Tuple{Vararg{V, N} where N}} where V
Tuple{typeof(convert),Type{Tuple{Vararg{V,N} where N}},Tuple{Vararg{V,N} where N}} where V

julia> struct Tracked{T}
       end

julia> A = Tuple{typeof(Base.convert), Type{T}, T} where T<:Tuple{Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}, Union{Nothing, Tracked}}
Tuple{typeof(convert),Type{T},T} where T<:NTuple{30,Union{Nothing, Tracked}}

julia> A<:B

JeffBezanson added a commit that referenced this issue Sep 21, 2019
@JeffBezanson
fix #33337 by reverting part of #32776 that changed method matching
71dbc3d
@JeffBezanson JeffBezanson mentioned this issue Sep 21, 2019
Closed
JeffBezanson added a commit that referenced this issue Sep 21, 2019
@JeffBezanson
add a subtyping fast path for tuple of Unions <: Vararg
3c2a6bd 
fixes #33337
@JeffBezanson JeffBezanson mentioned this issue Sep 21, 2019
Merged
JeffBezanson added a commit that referenced this issue Sep 21, 2019
@JeffBezanson
add a subtyping fast path for tuple of Unions <: Vararg
9dd7b5b 
fixes #33337
KristofferC pushed a commit that referenced this issue Sep 22, 2019
@JeffBezanson @KristofferC
add a subtyping fast path for tuple of Unions <: Vararg
2c757e7 
fixes #33337

(cherry picked from commit 9dd7b5b)
JeffBezanson added a commit that referenced this issue Sep 23, 2019
@JeffBezanson
add a subtyping fast path for tuple of Unions <: Vararg (#33353)
4f17558 
fixes #33337
JeffBezanson added a commit that referenced this issue Oct 1, 2019
@JeffBezanson
better fix for #33337; revert #33353
910c69c
JeffBezanson added a commit that referenced this issue Oct 2, 2019
@JeffBezanson
better fix for #33337; revert #33353 (#33440)
d911a9f
KristofferC pushed a commit that referenced this issue Oct 3, 2019
@JeffBezanson @KristofferC
better fix for #33337; revert #33353 (#33440)
68222cc 
(cherry picked from commit d911a9f)
@wookay wookay mentioned this issue Oct 13, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler:latency Compiler latency regression Regression in behavior compared to a previous version
Projects
None yet
Milestone
1.3
Development

Successfully merging a pull request may close this issue.

add a subtyping fast path for tuple of Unions <: Vararg JuliaLang/julia
2 participants
@JeffBezanson @KristofferC