UndefRefError with Dicts and finalizers

[amitmurthy]

  | | |_| | | | (_| |  |  Version 0.4.2 (2015-12-06 21:47 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-apple-darwin13.4.0

julia> const d = Dict()
Dict{Any,Any} with 0 entries

julia> type Foo
         a::Int
         b::Int
       end

julia> type Bar
         a::Int
         b::Int
       end

julia> function del_entry(id)
         global d
         delete!(d, id)
         nothing
       end
del_entry (generic function with 1 method)

julia> for n in 1:10^8
           f = Foo(n,n)
           finalizer(f, x->del_entry((x.a, x.b)))
           d[(n,n)] = Bar(n,n)
           yield()
       end
ERROR: UndefRefError: access to undefined reference
 in ht_keyindex2 at dict.jl:602
 in setindex! at dict.jl:643
 [inlined code] from none:4
 in anonymous at no file:0

I think this is the cause of #14295 as also the intermittent errors seen during the 0.4.2 release.
cc: @yuyichao , @tkelman

[tkelman]

Is it okay on master? Worth bisecting, on any branch it occurs.

[amitmurthy]

No, same error on master too.

[yuyichao]

The problem seems to be that the finalizer makes the Dict dirty during a GC in the middle of some Dict internal function and confuses it. Disabling the GC around d[(n, n)] = Bar(n, n) works around the problem in this simple case. We could probably add more check for h.dirty although it is also very hard to predict when will allocation happen.... (for example, it can happen when calling another helper function for non-leaftype key or value because that method haven't been compiled yet). Also ref #12157

[amitmurthy]

A workaround is changing delete!(d, id) to @async delete!(d, id). I'll submit a PR for this use of finalizers in the parallel code while this issue is being addressed.

[pearcemc]

Hi I was wondering if it is likely that the following error is caused by this same problem:

ERROR: UndefRefError: access to undefined reference
 [inlined code] from ./operators.jl:313
 in ht_keyindex at ./dict.jl:568
 in delete! at dict.jl:767
 in remotecall_wait at multi.jl:747
 [inlined code] from multi.jl:368
 in remotecall_wait at multi.jl:751
 in anonymous at task.jl:443

It's happening on a remote so I'm having a whale of a time trying to capture the source of the problem and find exactly what is failing. Thanks!

I'm on:

julia> versioninfo()
Julia Version 0.5.0-dev+749
Commit 83eac1e* (2015-10-13 16:00 UTC)

[amitmurthy]

Considering the workaround was merged on Dec 22 and your version is from Oct 13th, it probably is.

[amitmurthy]

This does not seem to be an issue anymore. Will revert #14456 once #15923 is fixed.