DictChannel not defined error

[tkelman]

Happens once in a while, I think I may have caused this actually in https://github.com/JuliaLang/julia/pull/15263/files#diff-58423af65cdd8c791e6fad72e0524984 - but why would myid() sometimes not return the right thing? Or is there a race condition?

example cases: https://ci.appveyor.com/project/JuliaLang/julia/build/1.0.792/job/sxyalbspyv27fcs2
https://ci.appveyor.com/project/JuliaLang/julia/build/1.0.776/job/jj6iyr4w22kdupi4
https://ci.appveyor.com/project/JuliaLang/julia/build/1.0.770/job/8ftjs48560jpbdh8
https://ci.appveyor.com/project/JuliaLang/julia/build/1.0.760/job/nm0icqna2eyrwpu8

[amitmurthy]

The change seems harmless.....I would suspect a race condition. Any recent changes to the code loading logic for Windows?

[amitmurthy]

Is there a way to search all AV logs under Julia for "LoadError: UndefVarError: DictChannel not defined" to find the earliest logged instance on this error?

[tkelman]

I believe so, there is an API reference at http://www.appveyor.com/docs/api/projects-builds though our history before we switched from the StefanKarpinski account to the JuliaLang account was unfortunately deleted.

[tkelman]

might have something to do with heavy load, it happened on an osx buildbot: https://build.julialang.org/builders/build_osx10.9-x64/builds/705/steps/shell_2/logs/stdio

[amitmurthy]

Unless code loading has an asynchronous part that completes after include returns, I don't see an obvious reason for this error to occur. I propose merging #16095 and seeing if it helps identify the cause.

[vtjnash]

I tried to queue a few more builds, but it looks like Appveyor's configuration is broken right now

[tkelman]

Doesn't include do something funny now in case the file gets modified while running? Nothing should be getting modified here but maybe the workaround for that is causing issues.

[vtjnash]

https://travis-ci.org/JuliaLang/julia/jobs/127403744

[tkelman]

should've merged #16095 a little earlier, that merge commit was on 0821d57 so just missed it

[vtjnash]

I suspect this may be caused by the same issue as #15923

[tkelman]

did you see this again? link?

[vtjnash]

yeah, locally

[tkelman]

@ranjanan saw this today in https://ci.appveyor.com/project/JuliaLang/julia/build/1.0.3375/job/tjrdj3erbwok88bm, looks like the new form this can sometimes take is MethodError: no method matching Base.Serializer.__deserialized_types__.##269()

[vtjnash]

Ah, this error makes far more sense. The old bug is probably that this conditional is sometimes wrong:

julia/base/serialize.jl

Lines 747 to 752 in a543e0c

	elseif isdefined(mod, name)
	tn = getfield(mod, name).name
	# TODO: confirm somehow that the types match
	name = tn.name
	mod = tn.module
	makenew = false

I spent a while trying to reproduce this, before I realized that Amit included a patch for this in

julia/base/clusterserialize.jl

Line 32 in a543e0c

elseif mod !== __deserialized_types__ && isdefined(mod, name)

which makes it far harder to trigger manually. This change was also my first thought to correct this, until I saw it was already being done that way and still failing.

Upon reflecting further, I realized this may only be half of the problem.

There seem to be a few candidates for the other half. For one, deserialize_cycle appears to be implemented incorrectly for TypeName (called too late), which could corrupt the backref list. For another, the assignment to known_object_data appears to also happen much too late, which might result in the creation of multiple TypeNames for a Type.

(A third issue, not encountered yet here due to the aforementioned bugs, but encountered previously by dump.c, the general deserializer for DataType.types could encounter instances of DataType before the call to jl_new_datatype, resulting in either the error error("Expected object in cache. Not found.") or equivalently, an Undef access of typename.primary)

[vtjnash]

https://ci.appveyor.com/project/JuliaLang/julia/build/1.0.4678/job/o2l19ya94nreab0j

min repro (in a fresh julia session):

julia> f = ()->1
(::#1) (generic function with 1 method)

julia> addprocs(1)
1-element Array{Int64,1}:
 2

julia> fetch(@spawnat 2 @eval f = ()->2)
(::#1) (generic function with 1 method)

julia> @spawnat 2 f()
Future(2,1,5,Nullable{Any}())

julia> fetch(ans) # should return 1
2

[amitmurthy]

This example is expected behavior. You have 2 different definitions for f on master and 2.

[JeffBezanson]

I think what's happening is that it tries to send over the local f, but the target machine sees it already has a function with the same name #1 so uses that instead. This might be the wrong thing to do for anonymous functions in Main.

[vtjnash]

While I used the same variable name, note that these are anonymous functions, not the function f

[amitmurthy]

I don't understand, isn't f bound to two different functions on master and 2?

@spawnat 2 f() actually executes an anonymous function ()->f() on 2, and it expects f to be already defined on 2, which it is.

[vtjnash]

Might be easier to see if you call them f1 and f2

[JeffBezanson]

@spawnat captures f from its environment. This example is clearer:

julia> f()
1

julia> remotecall_fetch(x->x(), 2, f)
2

I tried to explicitly send my f to processor 2, and it didn't work.

[amitmurthy]

It is even more confusing now.

On master:

julia> whos()
                          Base  36383 KB     Module
                          Core  12471 KB     Module
                          Main  43285 KB     Module
                           ans      0 bytes  Base.#whos
                            f1      0 bytes  ##1#2

On 2:

julia>  From worker 2:                            Base  34723 KB     Module
    From worker 2:                            Core  12352 KB     Module
    From worker 2:                            Main  41369 KB     Module
    From worker 2:                              f2      0 bytes  ##1#2

How can @spawnat 2 f1() ever return 2? It seems like it is being evaluated locally. remotecall_fetch has the correct behavior.

remotecall_fetch(()->f1(), 2)
ERROR: On worker 2:
UndefVarError: f1 not defined

[amitmurthy]

OTOH

julia> remotecall_fetch(f1, 2)
2

[amitmurthy]

@spawnat captures f from its environment.

My understanding was that globals are not captured. Guess the behavior is different for functions vs data?

[vtjnash]

I posted the min repro after seeing what needs to be added to my PR to fix this. The conditional at https://github.com/JuliaLang/julia/pull/17619/files#diff-aec7b7bfc3115e923e4137ad38876951R741
should have the following corrected branch structure:

elseif mod !== __deserialized_types__ && mod !== Main
    @assert isdefined(mod, name)
    ...
else
    mod = __deserialized_types__
    ....
end

[JeffBezanson]

My understanding was that globals are not captured.

julia> macroexpand(:(@spawnat 2 f()))
:((Base.spawnat)(2,(((f,)->(()->begin  # multi.jl, line 1658:
                    f()
                end)))(f)))

This behavior is a bit controversial though.

should have the following corrected branch structure

+1, though I think there's a chance we should only exclude Main for anonymous functions.

[vtjnash]

Right. That brings up the other possibility, which is that the serializer should redirect them into __deserialized_types__ instead of repeating the ishidden test in the deserializer. Any thoughts / preference?

[amitmurthy]

Ah! Just to reconfirm, this is due to Base.localize_vars localizing all variables, correct?

[JeffBezanson]

this is due to Base.localize_vars localizing all variables, correct

Yes.

[JeffBezanson]

serializer should redirect them into deserialized_types instead of repeating the ishidden test in the deserializer

Sounds like either would work.