UndefRefError with @threads for multithreading on 1.3

[Jutho]

I seem to get an error with some existing multithreaded code, using @threads for on 1.3. I don't think it is very deterministic, as I don't see it always, but the error does seem to originate in the code generated by the @threads macro, and not in my own code, i.e.

  UndefRefError: access to undefined reference
345  Stacktrace:
346   [1] try_yieldto(::typeof(Base.ensure_rescheduled), ::Base.RefValue{Task}) at ./task.jl:552
347   [2] wait() at ./task.jl:609
348   [3] wait(::Base.GenericCondition{Base.Threads.SpinLock}) at ./condition.jl:107
349   [4] wait(::Task) at ./task.jl:214

See a full report here:
https://travis-ci.org/Jutho/TensorKit.jl/builds/569731249
The error only occurs on 1.3 with multithreading, and all random numbers start from a common seed, so should the code that was ran should be the same on all platforms.

[JeffBezanson]

Try with #32814. Otherwise the stack trace from the original error is not visible.

[Jutho]

OK I'll give it a try; the problem is that it's not deterministic, the tests ran fine on 1.3 on my Mac.

[Jutho]

Thanks; that gives insight into the problem (which is my fault). Clearly, my first attempt at making LRUCache.jl was not very successful. I have close to zero experience with multithreaded programming, but I think I now fixed the problem. I'll close this here.

[kapple19]

I'm having this issue as well, just as you describe, it doesn't seem to be deterministic. I run the exact same code multiple times and the error pops up some of the runs, seemingly randomly.

This is my first experience with parallel programming as well. (Biting off a little more than I can chew.)

How did you fix it? I'm on 1.5.

[mzy2240]

I'm having similar issue as well, with version 1.8.2. Not deterministic as well, it tends to show up once a week. Since it is not deterministic, I am not able to give a MWE, but the pattern where the problem shows up is something like below:

container = []
Threads.@threads for (key, value) in collect(some_dict)
    push!(container, value)
end
res = vcat(container...)  # where UndefRefError shows up sometimes

[JeremiahLR]

Hello, I'm having the exact same problem in 1.9.1. Almost the same as mzy2240, @threads in a for loop and a push! (this time to a dataframe). But without the vcat. Error comes from the push! I think. Also not deterministic and only sometimes happens.

I think this issue should be reopened.

[oscardssmith]

I don't think this is a bug and is just that push! (and most operations on data-structures) in julia aren't thread-safe. you need a lock when resizing an array because otherwise multiple threads can try to resize the array at the same time.

[mzy2240]

@oscardssmith thanks for the explanation! Do you happen to have an example to show how @threads could work with lock? I tried but did not find any relevant example online.

[JeremiahLR]

Yes, an example of how to avoid/solve the issue would be most welcome.