Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mutex lock ordering problem between PDP and EDP classes [4627] #190

Closed
guillaumeautran opened this issue Mar 9, 2018 · 6 comments
Closed

Mutex lock ordering problem between PDP and EDP classes [4627] #190

guillaumeautran opened this issue Mar 9, 2018 · 6 comments

Comments

@guillaumeautran
Copy link
Contributor

There seems to be some mutex ordering problems between the EDP and PDP class such it causes a deadlock when a participant endpoint is deleted and a subscription to that endpoint is created at the same time.
@guillaumeautran
Copy link
Contributor Author

guillaumeautran commented Mar 9, 2018 •
edited
Loading

Hellgrind trace looks like this:

==18153== ----------------------------------------------------------------
==18153== 
==18153== Thread #15: lock order "0x16E56540 before 0x16E4AD70" violated
==18153== 
==18153== Observed (incorrect) order is: acquisition of lock at 0x16E4AD70
==18153==    at 0x4C321BC: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==18153==    by 0x8D7EBFF: __gthread_mutex_lock(pthread_mutex_t*) 
==18153==    by 0x8D7EC4F: __gthread_recursive_mutex_lock(pthread_mutex_t*) 
==18153==    by 0x8D7F5A7: std::recursive_mutex::lock() 
==18153==    by 0x8D7F7F7: std::lock_guard<std::recursive_mutex>::lock_guard(std::recursive_mutex&) 
==18153==    by 0x8E57FC6: eprosima::fastrtps::rtps::PDPSimple::removeReaderProxyData(eprosima::fastrtps::rtps::GUID_t const&) 
==18153==    by 0x8E6D887: eprosima::fastrtps::rtps::EDPSimple::removeLocalReader(eprosima::fastrtps::rtps::RTPSReader*) 
==18153==    by 0x8E55868: eprosima::fastrtps::rtps::BuiltinProtocols::removeLocalReader(eprosima::fastrtps::rtps::RTPSReader*) 
==18153==    by 0x8DFBB10: eprosima::fastrtps::rtps::RTPSParticipantImpl::deleteUserEndpoint(eprosima::fastrtps::rtps::Endpoint*) 
==18153==    by 0x8E01DC7: eprosima::fastrtps::rtps::RTPSDomain::removeRTPSReader(eprosima::fastrtps::rtps::RTPSReader*) 
==18153==    by 0x8E1ADB5: eprosima::fastrtps::SubscriberImpl::~SubscriberImpl() 
==18153==    by 0x8E1AE7F: eprosima::fastrtps::SubscriberImpl::~SubscriberImpl() 
==18153== 
==18153==  followed by a later acquisition of lock at 0x16E56540
==18153==    at 0x4C321BC: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==18153==    by 0x8D7EBFF: __gthread_mutex_lock(pthread_mutex_t*) 
==18153==    by 0x8D7EC4F: __gthread_recursive_mutex_lock(pthread_mutex_t*) 
==18153==    by 0x8D7F5A7: std::recursive_mutex::lock() 
==18153==    by 0x8D7F7F7: std::lock_guard<std::recursive_mutex>::lock_guard(std::recursive_mutex&) 
==18153==    by 0x8E6512D: eprosima::fastrtps::rtps::EDP::unpairReaderProxy(eprosima::fastrtps::rtps::GUID_t const&, eprosima::fastrtps::rtps::GUID_t const&) 
==18153==    by 0x8E5827F: eprosima::fastrtps::rtps::PDPSimple::removeReaderProxyData(eprosima::fastrtps::rtps::GUID_t const&) 
==18153==    by 0x8E6D887: eprosima::fastrtps::rtps::EDPSimple::removeLocalReader(eprosima::fastrtps::rtps::RTPSReader*) 
==18153==    by 0x8E55868: eprosima::fastrtps::rtps::BuiltinProtocols::removeLocalReader(eprosima::fastrtps::rtps::RTPSReader*) 
==18153==    by 0x8DFBB10: eprosima::fastrtps::rtps::RTPSParticipantImpl::deleteUserEndpoint(eprosima::fastrtps::rtps::Endpoint*) 
==18153==    by 0x8E01DC7: eprosima::fastrtps::rtps::RTPSDomain::removeRTPSReader(eprosima::fastrtps::rtps::RTPSReader*) 
==18153==    by 0x8E1ADB5: eprosima::fastrtps::SubscriberImpl::~SubscriberImpl() 
==18153== 
==18153== Required order was established by acquisition of lock at 0x16E56540
==18153==    at 0x4C321BC: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==18153==    by 0x8D7EBFF: __gthread_mutex_lock(pthread_mutex_t*) 
==18153==    by 0x8D7EC4F: __gthread_recursive_mutex_lock(pthread_mutex_t*) 
==18153==    by 0x8D7F5A7: std::recursive_mutex::lock() 
==18153==    by 0x8D7F7F7: std::lock_guard<std::recursive_mutex>::lock_guard(std::recursive_mutex&) 
==18153==    by 0x8E68BA9: eprosima::fastrtps::rtps::EDP::pairing_reader_proxy_with_any_local_writer(eprosima::fastrtps::rtps::ParticipantProxyData*, eprosima::fastrtps::rtps::ReaderProxyData*) 
==18153==    by 0x8E6422E: eprosima::fastrtps::rtps::EDP::newLocalReaderProxyData(eprosima::fastrtps::rtps::RTPSReader*, eprosima::fastrtps::TopicAttributes&, eprosima::fastrtps::ReaderQos&) 
==18153==    by 0x8E554D6: eprosima::fastrtps::rtps::BuiltinProtocols::addLocalReader(eprosima::fastrtps::rtps::RTPSReader*, eprosima::fastrtps::TopicAttributes&, eprosima::fastrtps::ReaderQos&) 
==18153==    by 0x8DFA8FA: eprosima::fastrtps::rtps::RTPSParticipantImpl::registerReader(eprosima::fastrtps::rtps::RTPSReader*, eprosima::fastrtps::TopicAttributes&, eprosima::fastrtps::ReaderQos&) 
==18153==    by 0x8DF5CD5: eprosima::fastrtps::rtps::RTPSParticipant::registerReader(eprosima::fastrtps::rtps::RTPSReader*, eprosima::fastrtps::TopicAttributes&, eprosima::fastrtps::ReaderQos&) 
==18153==    by 0x8E0A58E: eprosima::fastrtps::ParticipantImpl::createSubscriber(eprosima::fastrtps::SubscriberAttributes&, eprosima::fastrtps::SubscriberListener*) 
==18153==    by 0x8E045B5: eprosima::fastrtps::Domain::createSubscriber(eprosima::fastrtps::Participant*, eprosima::fastrtps::SubscriberAttributes&, eprosima::fastrtps::SubscriberListener*) 
==18153== 
==18153==  followed by a later acquisition of lock at 0x16E4AD70
==18153==    at 0x4C321BC: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==18153==    by 0x8D7EBFF: __gthread_mutex_lock(pthread_mutex_t*) 
==18153==    by 0x8D7EC4F: __gthread_recursive_mutex_lock(pthread_mutex_t*) 
==18153==    by 0x8D7F5A7: std::recursive_mutex::lock() 
==18153==    by 0x8D7F7F7: std::lock_guard<std::recursive_mutex>::lock_guard(std::recursive_mutex&) 
==18153==    by 0x8E57DFB: eprosima::fastrtps::rtps::PDPSimple::lookupWriterProxyData(eprosima::fastrtps::rtps::GUID_t const&, eprosima::fastrtps::rtps::WriterProxyData&, eprosima::fastrtps::rtps::ParticipantProxyData&) 
==18153==    by 0x8E68CB7: eprosima::fastrtps::rtps::EDP::pairing_reader_proxy_with_any_local_writer(eprosima::fastrtps::rtps::ParticipantProxyData*, eprosima::fastrtps::rtps::ReaderProxyData*) 
==18153==    by 0x8E6422E: eprosima::fastrtps::rtps::EDP::newLocalReaderProxyData(eprosima::fastrtps::rtps::RTPSReader*, eprosima::fastrtps::TopicAttributes&, eprosima::fastrtps::ReaderQos&) 
==18153==    by 0x8E554D6: eprosima::fastrtps::rtps::BuiltinProtocols::addLocalReader(eprosima::fastrtps::rtps::RTPSReader*, eprosima::fastrtps::TopicAttributes&, eprosima::fastrtps::ReaderQos&) 
==18153==    by 0x8DFA8FA: eprosima::fastrtps::rtps::RTPSParticipantImpl::registerReader(eprosima::fastrtps::rtps::RTPSReader*, eprosima::fastrtps::TopicAttributes&, eprosima::fastrtps::ReaderQos&) 
==18153==    by 0x8DF5CD5: eprosima::fastrtps::rtps::RTPSParticipant::registerReader(eprosima::fastrtps::rtps::RTPSReader*, eprosima::fastrtps::TopicAttributes&, eprosima::fastrtps::ReaderQos&) 
==18153==    by 0x8E0A58E: eprosima::fastrtps::ParticipantImpl::createSubscriber(eprosima::fastrtps::SubscriberAttributes&, eprosima::fastrtps::SubscriberListener*) 
==18153== 
==18153==  Lock at 0x16E56540 was first observed
==18153==    at 0x4C321BC: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==18153==    by 0x8D7EBFF: __gthread_mutex_lock(pthread_mutex_t*) 
==18153==    by 0x8D7EC4F: __gthread_recursive_mutex_lock(pthread_mutex_t*) 
==18153==    by 0x8D7F5A7: std::recursive_mutex::lock() 
==18153==    by 0x8D7F7F7: std::lock_guard<std::recursive_mutex>::lock_guard(std::recursive_mutex&) 
==18153==    by 0x8DF9AAF: eprosima::fastrtps::rtps::RTPSParticipantImpl::createWriter(eprosima::fastrtps::rtps::RTPSWriter**, eprosima::fastrtps::rtps::WriterAttributes&, eprosima::fastrtps::rtps::WriterHistory*, eprosima::fastrtps::rtps::WriterListener*, eprosima::fastrtps::rtps::EntityId_t const&, bool) 
==18153==    by 0x8E58D30: eprosima::fastrtps::rtps::PDPSimple::createSPDPEndpoints() 
==18153==    by 0x8E5681D: eprosima::fastrtps::rtps::PDPSimple::initPDP(eprosima::fastrtps::rtps::RTPSParticipantImpl*) 
==18153==    by 0x8E54F17: eprosima::fastrtps::rtps::BuiltinProtocols::initBuiltinProtocols(eprosima::fastrtps::rtps::RTPSParticipantImpl*, eprosima::fastrtps::rtps::BuiltinAttributes&) 
==18153==    by 0x8DF8889: eprosima::fastrtps::rtps::RTPSParticipantImpl::RTPSParticipantImpl(eprosima::fastrtps::rtps::RTPSParticipantAttributes const&, eprosima::fastrtps::rtps::GuidPrefix_t const&, eprosima::fastrtps::rtps::RTPSParticipant*, eprosima::fastrtps::rtps::RTPSParticipantListener*) 
==18153==    by 0x8E01434: eprosima::fastrtps::rtps::RTPSDomain::createParticipant(eprosima::fastrtps::rtps::RTPSParticipantAttributes&, eprosima::fastrtps::rtps::RTPSParticipantListener*) 
==18153==    by 0x8E03DA0: eprosima::fastrtps::Domain::createParticipant(eprosima::fastrtps::ParticipantAttributes&, eprosima::fastrtps::ParticipantListener*) 
==18153==  Address 0x16e56540 is 0 bytes inside a block of size 40 alloc'd
==18153==    at 0x4C2F50F: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==18153==    by 0x8DF7996: eprosima::fastrtps::rtps::RTPSParticipantImpl::RTPSParticipantImpl(eprosima::fastrtps::rtps::RTPSParticipantAttributes const&, eprosima::fastrtps::rtps::GuidPrefix_t const&, eprosima::fastrtps::rtps::RTPSParticipant*, eprosima::fastrtps::rtps::RTPSParticipantListener*) 
==18153==    by 0x8E01434: eprosima::fastrtps::rtps::RTPSDomain::createParticipant(eprosima::fastrtps::rtps::RTPSParticipantAttributes&, eprosima::fastrtps::rtps::RTPSParticipantListener*) 
==18153==    by 0x8E03DA0: eprosima::fastrtps::Domain::createParticipant(eprosima::fastrtps::ParticipantAttributes&, eprosima::fastrtps::ParticipantListener*) 
==18153==  Block was alloc'd by thread #1
==18153== 
==18153==  Lock at 0x16E4AD70 was first observed
==18153==    at 0x4C321BC: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==18153==    by 0x8D7EBFF: __gthread_mutex_lock(pthread_mutex_t*) 
==18153==    by 0x8D7EC4F: __gthread_recursive_mutex_lock(pthread_mutex_t*) 
==18153==    by 0x8D7F5A7: std::recursive_mutex::lock() 
==18153==    by 0x8E5713B: eprosima::fastrtps::rtps::PDPSimple::announceParticipantState(bool, bool) 
==18153==    by 0x8E55074: eprosima::fastrtps::rtps::BuiltinProtocols::initBuiltinProtocols(eprosima::fastrtps::rtps::RTPSParticipantImpl*, eprosima::fastrtps::rtps::BuiltinAttributes&) 
==18153==    by 0x8DF8889: eprosima::fastrtps::rtps::RTPSParticipantImpl::RTPSParticipantImpl(eprosima::fastrtps::rtps::RTPSParticipantAttributes const&, eprosima::fastrtps::rtps::GuidPrefix_t const&, eprosima::fastrtps::rtps::RTPSParticipant*, eprosima::fastrtps::rtps::RTPSParticipantListener*) 
==18153==    by 0x8E01434: eprosima::fastrtps::rtps::RTPSDomain::createParticipant(eprosima::fastrtps::rtps::RTPSParticipantAttributes&, eprosima::fastrtps::rtps::RTPSParticipantListener*) 
==18153==    by 0x8E03DA0: eprosima::fastrtps::Domain::createParticipant(eprosima::fastrtps::ParticipantAttributes&, eprosima::fastrtps::ParticipantListener*) 
==18153==  Block was alloc'd by thread #1
==18153== 
==18153== 
==18153== ----------------------------------------------------------------

@sagniknitr
Copy link

@guillaumeautran Hi. Can you mention the steps to reproduce this problem ?

@guillaumeautran
Copy link
Contributor Author

Yes, this issue will happen when a new subscriber is created locally while at the same time the participant gets un-registered.
When that happens, the stack deadlocks.

@sagniknitr sagniknitr mentioned this issue Mar 16, 2018
Open
@matt-attack
Copy link

I have noticed deadlocks in fastrtps like this as well when trying to setup multiple nodes in the same process. Nice find!

@guillaumeautran guillaumeautran mentioned this issue Mar 19, 2018
Closed
@guillaumeautran guillaumeautran mentioned this issue Jun 21, 2018
Closed
guillaumeautran pushed a commit to clearpathrobotics/Fast-RTPS that referenced this issue Jun 27, 2018
@deng02 @guillaumeautran
Use the interface whitelist to bind multicast input sockets and filte…
96544ad 
…r locators.

The UDP descriptors have been modified to support flags that
allow the interface white list to be applied to output sockets,
multicast input sockets and/or locators.  By default the white
list is only applied to output sockets.

If applied to multicast input sockets, rather than binding to IP address
any (0.0.0.0), the input socket will bind to specific IPs.

If applied to locators, then the default unicast locator list
will be filtered against the allowed interface IPs.  It is not
applied in the case where the user has specified a default list.

The XML profile has been modified tosupport configuring the
interface white list via the user transport tag.

Issue: eProsima#190
JuanCarlos-Arce pushed a commit to clearpathrobotics/Fast-RTPS that referenced this issue Oct 16, 2018
@deng02
Use the interface whitelist to bind multicast input sockets and filte…
2596ee7 
…r locators.

The UDP descriptors have been modified to support flags that
allow the interface white list to be applied to output sockets,
multicast input sockets and/or locators.  By default the white
list is only applied to output sockets.

If applied to multicast input sockets, rather than binding to IP address
any (0.0.0.0), the input socket will bind to specific IPs.

If applied to locators, then the default unicast locator list
will be filtered against the allowed interface IPs.  It is not
applied in the case where the user has specified a default list.

The XML profile has been modified tosupport configuring the
interface white list via the user transport tag.

Issue: eProsima#190
JuanCarlos-Arce pushed a commit that referenced this issue Oct 25, 2018
@deng02
Use the interface whitelist to bind multicast input sockets and filte…
78f5f7e 
…r locators.

The UDP descriptors have been modified to support flags that
allow the interface white list to be applied to output sockets,
multicast input sockets and/or locators.  By default the white
list is only applied to output sockets.

If applied to multicast input sockets, rather than binding to IP address
any (0.0.0.0), the input socket will bind to specific IPs.

If applied to locators, then the default unicast locator list
will be filtered against the allowed interface IPs.  It is not
applied in the case where the user has specified a default list.

The XML profile has been modified tosupport configuring the
interface white list via the user transport tag.

Issue: #190
JuanCarlos-Arce pushed a commit that referenced this issue Oct 25, 2018
@deng02
Use the interface whitelist to bind multicast input sockets and filte…
a656db7 
…r locators.

The UDP descriptors have been modified to support flags that
allow the interface white list to be applied to output sockets,
multicast input sockets and/or locators.  By default the white
list is only applied to output sockets.

If applied to multicast input sockets, rather than binding to IP address
any (0.0.0.0), the input socket will bind to specific IPs.

If applied to locators, then the default unicast locator list
will be filtered against the allowed interface IPs.  It is not
applied in the case where the user has specified a default list.

The XML profile has been modified tosupport configuring the
interface white list via the user transport tag.

Issue: #190
@raquelalvarezbanos
Copy link
Contributor

Hi @guillaumeautran I was reviewing this old issue, what is the status of it? Can it be closed?

@richiware richiware changed the title Mutex lock ordering problem between PDP and EDP classes Mutex lock ordering problem between PDP and EDP classes [4627] Feb 11, 2019
@guillaumeautran
Copy link
Contributor Author

I have not seen this issue since the interface whitelisting feature went in. So yes, we can probably close it and if it re-appears, open a new issue. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@matt-attack @raquelalvarezbanos @guillaumeautran @sagniknitr