Occasional async readback deadlocks on GLX #177

hernando · 2012-11-13T15:35:14Z

Part of the backtrace of the aborting thread in the app:
...
#5 0x00007fa411f8c101 in eq::Image::~Image (this=0x7fa4040324e0, __in_chrg=)

at /home/jhernando/bbp/Buildyard/src/Equalizer/libs/eq/client/image.cpp:228

#6 0x00007fa411f8c144 in eq::Image::~Image (this=0x7fa4040324e0, __in_chrg=)

at /home/jhernando/bbp/Buildyard/src/Equalizer/libs/eq/client/image.cpp:229

#7 0x00007fa411f82e1a in eq::FrameData::~FrameData (this=0x14b9eb0, __in_chrg=)

at /home/jhernando/bbp/Buildyard/src/Equalizer/libs/eq/client/frameData.cpp:67

#8 0x00007fa411f8305e in eq::FrameData::~FrameData (this=0x14b9eb0, __in_chrg=)

at /home/jhernando/bbp/Buildyard/src/Equalizer/libs/eq/client/frameData.cpp:73

...
#18 0x00007fa411fd8790 in eq::Window::_cmdDestroyChannel (this=0x7fa40402cd00, cmd=...)

at /home/jhernando/bbp/Buildyard/src/Equalizer/libs/eq/client/window.cpp:791

...

The rendering client closes it's pipe, but stays alive after the crash.

The text was updated successfully, but these errors were encountered:

hernando · 2012-11-13T15:56:00Z

https://gist.github.com/4066441#file_app_node.trace
https://gist.github.com/4066441#file_client.trace
https://gist.github.com/4066441#file_config.eqc
https://gist.github.com/4066441#file_app_node.log

I lost the original app node log. The log above is from a different run

eile · 2012-11-13T16:26:29Z

I solved a couple of these for 1.4 already, will have a look at it.

hernando · 2012-11-13T16:31:59Z

It also happens with RTNeuron but only with DB compounds. I've commented the override of eq::Channel::frameAssemble on both and then they don't segfault anymore.

eile · 2012-11-13T16:47:53Z

The segfaults above are from an assertion which should be harmless in release mode. I'll fix this one in any case, hopefully tomorrow.

eile · 2012-11-14T09:40:02Z

@hernando: Please verify fix and close bug

hernando · 2012-11-14T11:14:37Z

4 out of 10 tries deadlocked, another one aborted.
For the deadlocks they all look the same:
https://gist.github.com/4066441#file_app_node.deadlock.trace
https://gist.github.com/4066441#file_app_node.deadlock.log
https://gist.github.com/4066441#file_client.deadlock.trace
https://gist.github.com/4066441#file_client.deadlock.log

For the aborted one I only have the trace:
https://gist.github.com/4066441#file_app_node.aborted.log

eile · 2012-11-14T13:03:38Z

Ok, that's a different one I didn't see. Investigating...

eile · 2012-11-14T13:14:06Z

The crash seems harmless, it's just a sanity check for reference counting/deletion.

The deadlock happens here, within my favourite driver:

Thread 2 (Thread 0x7f37d78ff700 (LWP 10088)):
#0  0x00007f37e3a6ae23 in __GI___poll (fds=<optimized out>, 
    nfds=<optimized out>, timeout=<optimized out>)
    at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00007f37df9424f2 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#2  0x00007f37df943aaf in xcb_wait_for_reply ()
   from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#3  0x00007f37e3441b7d in _XReply () from /usr/lib/x86_64-linux-gnu/libX11.so.6
#4  0x00007f37e532b3b1 in ?? () from /usr/lib/libGL.so.1
#5  0x00007f37e52fdc38 in ?? () from /usr/lib/libGL.so.1
#6  0x00007f37e52f2bfc in glXCreateNewContext () from /usr/lib/libGL.so.1
#7  0x00007f37e57cd36a in eq::glx::Window::createGLXContext (this=0x1182c50,

This trace: https://gist.github.com/4066441#file_client.deadlock.trace

eile · 2012-11-14T13:20:45Z

This is again async readback related. Can you reproduce it with async rb off?

hernando · 2012-11-14T13:35:20Z

On 14/11/12 14:20, Stefan Eilemann wrote:

This is again async readback related. Can you reproduce it with async rb off?
How do I do it?
I've skimed over the compilation options with ccmake and taken a look at
loader.l, but I haven't found how to do it.

eile · 2012-11-14T13:41:06Z

Remove EQ_COMPRESSOR_USE_ASYNC_DOWNLOAD from compressorReadDrawPixels.cpp

Daniel and me have a hypothesis:

That behavior is expected, none of the contexts in the wglShareLists call can be current in a different thread to the one you are calling wglShareLists from. That important piece of information is missing from the MSDN. [http://www.opengl.org/discussion_boards/showthread.php/152648-wglShareLists-failing]

The glx context sharing likely has a similar constraint.

… thread instead of transfer thread

hernando · 2012-11-14T14:43:54Z

After disabling async readback the test doesn't deadlock anymore. The original issue can be considered solved.

eile · 2012-11-14T15:01:12Z

I respectfully disagree, the async readback should work.

Did you get a chance to test 39e45bf with async readbacks on?

hernando · 2012-11-14T15:05:35Z

I was just saying that the issue originally reported has been solved and this ticket as diverted into a different issue, shouldn't it be renamed then?

hernando · 2012-11-14T15:52:33Z

Now it's unlikely and slightly different:
I run it more than a dozen times and a couple I got this
https://gist.github.com/4066441#file_app_node.deadlock.trace.2
https://gist.github.com/4066441#file_client.deadlock.trace.2

#0 0x00007f497bd95e23 in __GI___poll (fds=,
    nfds=, timeout=)
    at ../sysdeps/unix/sysv/linux/poll.c:87
#1 0x00007f4977c6d4f2 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#2 0x00007f4977c6eaaf in xcb_wait_for_reply ()
   from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#3 0x00007f497b76cb7d in _XReply () from /usr/lib/x86_64-linux-gnu/libX11.so.6
#4 0x00007f497d6287d8 in ?? () from /usr/lib/libGL.so.1
#5 0x00007f497d629bab in ?? () from /usr/lib/libGL.so.1
#6 0x00007f497daf9873 in eq::glx::Window::makeCurrent (this=0x7f49703cef40,
    cache=true)
    at /home/jhernando/bbp/Buildyard/src/Equalizer/libs/eq/client/glx/window.cpp:709
#7 0x00007f497dae40c7 in eq::Window::makeCurrentTransfer (
    this=0x7f497019bf90, useCache=true)
    at /home/jhernando/bbp/Buildyard/src/Equalizer/libs/eq/client/window.cpp:528
#8 0x00007f497da45e0f in eq::Channel::_cmdFinishReadback (
    this=0x7f4970337340, cmd=...)
    at /home/jhernando/bbp/Buildyard/src/Equalizer/libs/eq/client/channel.cpp:2164

eile · 2012-11-14T18:56:58Z

The deadlock is:

Thread A: glXSwapBuffers ... __lll_lock_wait
Thread B: eq::glx::Window::makeCurrent ... _XReply ... __GI___poll

This smells not good. Right now the only option I can think of is to downgrade the drivers to 270. XInitThreads is called, so I don't see what we're doing wrong here.

eile · 2012-11-14T18:58:22Z

Do you see similar issues in RTNeuron/eVolve with multiprocess mode?

hernando · 2012-11-14T19:35:42Z

This is indeed a multiprocess configuration, the rendering client and application node are separate processes, and I've also seen the issue with RTNeuron.

Another important detail is that the tests have been done in a single GPU machine, I haven't tested in multi-GPU.

eile · 2012-11-14T19:40:23Z

Ah, the two 'GPU' threads are render and transfer. Bummer.

… thread instead of transfer thread Conflicts: libs/eq/client/channel.cpp libs/eq/client/pipe.cpp

transfer window context temporarily current in draw thread It looks like the driver realizes the context on the first makeCurrent (hmm, reverse engineering), still observing the deadlock occasionally. This 'useless' makecurrent hopefully solves this issue.

eile · 2012-11-16T14:05:21Z

@hernando: Please test again whenever it's convenient.

hernando · 2012-11-17T00:54:32Z

Tried again (around 25 times)
Once got a different deadlock. This seems to be an unrelated driver issue.

#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1  0x00007fe62d628200 in _L_lock_928 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007fe62d628099 in __pthread_mutex_lock (mutex=0x7fe62e6cc860) at pthread_mutex_lock.c:82
#3  0x00007fe62e46a042 in ?? () from /usr/lib/libGL.so.1
#4  0x00007fe62e443217 in ?? () from /usr/lib/libGL.so.1
#5  0x00007fe62e439013 in glXSwapBuffers () from /usr/lib/libGL.so.1
#6  0x00007fe62e916b07 in eq::glx::Window::swapBuffers (this=0x11c8b40)
    at /home/jhernando/bbp/Buildyard/src/Equalizer/libs/eq/client/glx/window.cpp:720
#7  0x00007fe62e9014db in eq::Window::swapBuffers (this=0x13864c0)
    at /home/jhernando/bbp/Buildyard/src/Equalizer/libs/eq/client/window.cpp:591
#8  0x000000000045fcbd in eVolve::Window::swapBuffers (this=0x13864c0)
    at /home/jhernando/bbp/Buildyard/src/Equalizer/examples/eVolve/window.cpp:144

And one seemingly harmless abort at exit (I don't have full backtraces for this one)

32056 Main /home/jhernando/bbp/Buildyard/src/Collage/co/commandQueue.cpp:52 5149 Flushing non-empty c
ommand queue
32056 Main /home/jhernando/bbp/Buildyard/src/Collage/co/localNode.cpp:448 5149 Assert: connection->ge
tRefCount()==2 || connection->getDescription()->type >= co::CONNECTIONTYPE_MULTICAST [3: Connection 0
x7f71fc01d3c0 type N2co16SocketConnectionE state closed description TCPIP#102400#localhost##45168#def
ault#] , in: 
    lunchbox::abort()
    co::LocalNode::removeListeners(std::vector, std::allocator > > const&)
    eq::Config::notifyDetach()
    co::ObjectStore::unmapObject(co::Object*)
    co::LocalNode::unmapObject(co::Object*)
    eq::fabric::Server, eq::fabric::ElementVisitor >, eq::fabric::ElementVisitor >, eq::fabric::ElementVisitor > > > > > >::_cmdDestroyConfig(co::ICommand&)
    co::CommandFunc::operator()(co::ICommand&)
    co::ICommand::operator()()
    eq::fabric::Client::processCommand(unsigned int)
    eq::Server::releaseConfig(eq::Config*)
    eVolve::EVolve::run()
    ../../Equalizer/bin/eVolve(main+0x1c0) [0x456e70]
    __libc_start_main

eile · 2012-11-26T12:51:09Z

The remaining deadlock looks like a driver issue. Will move to 1.6 milestone.

…ale#177; create shared context from render thread instead of async fetch thread

eile · 2013-01-22T10:00:53Z

@hernando: While testing the 270/310 drivers, can you also report if this issue is reproducible?

hernando · 2013-01-24T18:12:01Z

I've tested eVolve more than 30 times with 310.32 and I couldn't reproduce it.
By the way, what happened to the rendering client log files?

eile · 2013-01-25T08:42:59Z

Closing, seems like a driver issue.

ghost assigned eile Nov 13, 2012

eile pushed a commit that referenced this issue Nov 14, 2012

eVolve: properly dispose of "self" assembly frame (#177)

f3951c4

This was referenced Nov 14, 2012

Windows: PBO error when rendering #138

Closed

PixelBufferObject (glMapBuffer) rendering error in RHEL #174

Closed

eile pushed a commit that referenced this issue Nov 14, 2012

Potential fix for #138, #174, #177: Create shared context from render…

39e45bf

… thread instead of transfer thread

hernando closed this as completed Nov 14, 2012

eile reopened this Nov 14, 2012

eile pushed a commit that referenced this issue Nov 14, 2012

Potential fix for #138, #174, #177: Create shared context from render…

e772e14

… thread instead of transfer thread Conflicts: libs/eq/client/channel.cpp libs/eq/client/pipe.cpp

tribal-tec added a commit to tribal-tec/Equalizer that referenced this issue Nov 26, 2012

Fix Eyescale#183: Same issue as for Eyescale#138, Eyescale#174, Eyesc…

2168835

…ale#177; create shared context from render thread instead of async fetch thread

hernando closed this as completed Jan 23, 2013

hernando reopened this Jan 23, 2013

eile closed this as completed Jan 25, 2013

tribal-tec mentioned this issue Aug 5, 2015

Fix transfer window deadlock with Qt, implement Window::doneCurrent() #484

Merged

eile mentioned this issue Nov 10, 2015

Fix #510: Async readback deadlock with direct send compositing #511

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Occasional async readback deadlocks on GLX #177

Occasional async readback deadlocks on GLX #177

hernando commented Nov 13, 2012

hernando commented Nov 13, 2012

eile commented Nov 13, 2012

hernando commented Nov 13, 2012

eile commented Nov 13, 2012

eile commented Nov 14, 2012

hernando commented Nov 14, 2012

eile commented Nov 14, 2012

eile commented Nov 14, 2012

eile commented Nov 14, 2012

hernando commented Nov 14, 2012

eile commented Nov 14, 2012

hernando commented Nov 14, 2012

eile commented Nov 14, 2012

hernando commented Nov 14, 2012

hernando commented Nov 14, 2012

eile commented Nov 14, 2012

eile commented Nov 14, 2012

hernando commented Nov 14, 2012

eile commented Nov 14, 2012

eile commented Nov 16, 2012

hernando commented Nov 17, 2012

eile commented Nov 26, 2012

eile commented Jan 22, 2013

hernando commented Jan 24, 2013

eile commented Jan 25, 2013

Occasional async readback deadlocks on GLX #177

Occasional async readback deadlocks on GLX #177

Comments

hernando commented Nov 13, 2012

hernando commented Nov 13, 2012

eile commented Nov 13, 2012

hernando commented Nov 13, 2012

eile commented Nov 13, 2012

eile commented Nov 14, 2012

hernando commented Nov 14, 2012

eile commented Nov 14, 2012

eile commented Nov 14, 2012

eile commented Nov 14, 2012

hernando commented Nov 14, 2012

eile commented Nov 14, 2012

hernando commented Nov 14, 2012

eile commented Nov 14, 2012

hernando commented Nov 14, 2012

hernando commented Nov 14, 2012

eile commented Nov 14, 2012

eile commented Nov 14, 2012

hernando commented Nov 14, 2012

eile commented Nov 14, 2012

eile commented Nov 16, 2012

hernando commented Nov 17, 2012

eile commented Nov 26, 2012

eile commented Jan 22, 2013

hernando commented Jan 24, 2013

eile commented Jan 25, 2013