Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leaks: Error in destruction of rcl action client handle: the Node Handle was destructed too early. #1545

Closed
zhoulaifu opened this issue Feb 7, 2021 · 20 comments
Labels
bug Something isn't working more-information-needed Further information is required

Comments

@zhoulaifu
Copy link

Memory leaks are found when running test_buffer_client of tf2_ros. with the Address Sanitizer (https://en.wikipedia.org/wiki/AddressSanitizer) on.

Below is the error message. Let me know if you need a dockerfile to reproduce it.

root@5490e71bc767:/opt/ros_ws# ./build/tf2_ros/test_buffer_client
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from TestBufferClient
[ RUN      ] TestBufferClient.lookup_transform_available
[       OK ] TestBufferClient.lookup_transform_available (236 ms)
[ RUN      ] TestBufferClient.lookup_transform_unavailable
[       OK ] TestBufferClient.lookup_transform_unavailable (175 ms)
[ RUN      ] TestBufferClient.can_transform_available
[       OK ] TestBufferClient.can_transform_available (172 ms)
[ RUN      ] TestBufferClient.can_transform_unavailable
[ERROR] [1612723447.179594300] [rclcpp_action]: Error in destruction of rcl action client handle: the Node Handle was destructed too early. You will leak memory
[       OK ] TestBufferClient.can_transform_unavailable (170 ms)
[----------] 4 tests from TestBufferClient (753 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (823 ms total)
[  PASSED  ] 4 tests.

=================================================================
==30==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 568 byte(s) in 1 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055cc2d2da in rcl_action_client_init (/opt/ros/foxy/lib/librcl_action.so+0x52da)

Direct leak of 56 byte(s) in 1 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c635892  (/opt/ros/foxy/lib/librcl.so+0x15892)
    #2 0x7f055cd1dd0a in rclcpp::graph_listener::GraphListener::GraphListener(std::shared_ptr<rclcpp::Context>) (/opt/ros/foxy/lib/librclcpp.so+0xe2d0a)
    #3 0x7f055cd4181b in std::shared_ptr<rclcpp::graph_listener::GraphListener> rclcpp::Context::get_sub_context<rclcpp::graph_listener::GraphListener, std::shared_ptr<rclcpp::Context> >(std::shared_ptr<rclcpp::Context>&&) (/opt/ros/foxy/lib/librclcpp.so+0x10681b)

Indirect leak of 480 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c6412d8 in rcl_subscription_init (/opt/ros/foxy/lib/librcl.so+0x212d8)

Indirect leak of 456 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d4708 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1c708)

Indirect leak of 432 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c63113e in rcl_client_init (/opt/ros/foxy/lib/librcl.so+0x1113e)

Indirect leak of 384 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d3026 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1b026)

Indirect leak of 384 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597e2e60  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2ae60)
    #2 0x7f05597e0de1 in rmw_create_subscription (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x28de1)

Indirect leak of 360 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d4d20 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1cd20)

Indirect leak of 360 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d3110 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1b110)

Indirect leak of 240 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597e2d18  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2ad18)
    #2 0x7f05597e0de1 in rmw_create_subscription (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x28de1)

Indirect leak of 201 byte(s) in 6 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c5ea56f in rmw_allocate (/opt/ros/foxy/lib/librmw.so+0x256f)

Indirect leak of 192 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d4eb0 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1ceb0)

Indirect leak of 164 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f05597e466e  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2c66e)

Indirect leak of 161 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f05597e43de  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2c3de)

Indirect leak of 160 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597e1ec5  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x29ec5)
    #2 0x7f05597e0de1 in rmw_create_subscription (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x28de1)

Indirect leak of 103 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055cbef37c in rcutils_strndup (/opt/ros/foxy/lib/librcutils.so+0xd37c)

Indirect leak of 98 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f05597e4156  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2c156)

Indirect leak of 96 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c5ea56f in rmw_allocate (/opt/ros/foxy/lib/librmw.so+0x256f)
    #2 0x7f05597e0de1 in rmw_create_subscription (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x28de1)

Indirect leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f055978faf7 in rmw_fastrtps_shared_cpp::__rmw_create_guard_condition(char const*) (/opt/ros/foxy/lib/librmw_fastrtps_shared_cpp.so+0x2caf7)

Indirect leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f055978fae7 in rmw_fastrtps_shared_cpp::__rmw_create_guard_condition(char const*) (/opt/ros/foxy/lib/librmw_fastrtps_shared_cpp.so+0x2cae7)

SUMMARY: AddressSanitizer: 4983 byte(s) leaked in 49 allocation(s).
@fujitatomoya
Copy link
Collaborator

@zhoulaifu

could you share the complete procedure to reproduce this issue? i want to check if this still happens in source build.

@fujitatomoya fujitatomoya added the more-information-needed Further information is required label Feb 8, 2021
@zhoulaifu
Copy link
Author

You can use this dockerfile to reproduce the environmental setting.

In the docker container, you have a rox:foxy system with geometry2
compiled with the Address Sanitizer (ASan). Then run the following
binary from the tf2_ros package to reproduce the bug.
${ROS_WS}/build/tf2_ros/test_buffer_client

Please let me know if anything unclear.

@fujitatomoya
Copy link
Collaborator

memory leak confirmed with source build ros2/ros2@9552559

colcon_ws# VERBOSE=1 CXX="g++" CXXFLAGS="--coverage -g -fsanitize=address" colcon build --event-handlers console_direct+ --packages-select tf2_ros
colcon_ws# ./build/tf2_ros/test_buffer_client
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from TestBufferClient
[ RUN      ] TestBufferClient.lookup_transform_available
[ERROR] [1612790271.217678208] [rclcpp_action]: Error in destruction of rcl action client handle: the Node Handle was destructed too early. You will leak memory
[       OK ] TestBufferClient.lookup_transform_available (124 ms)
[ RUN      ] TestBufferClient.lookup_transform_unavailable
[       OK ] TestBufferClient.lookup_transform_unavailable (119 ms)
[ RUN      ] TestBufferClient.can_transform_available
[       OK ] TestBufferClient.can_transform_available (118 ms)
[ RUN      ] TestBufferClient.can_transform_unavailable
[       OK ] TestBufferClient.can_transform_unavailable (120 ms)
[----------] 4 tests from TestBufferClient (481 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (484 ms total)
[  PASSED  ] 4 tests.

@fujitatomoya
Copy link
Collaborator

could be similar with #1296

@fujitatomoya
Copy link
Collaborator

just sharing debug information.

stack trace via gdb
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f5c9d91e859 in __GI_abort () at abort.c:79
#2  0x00007f5c9e780eab in rclcpp_action::ClientBaseImpl::ClientBaseImpl(std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>, std::shared_ptr<rclcpp::node_interfaces::NodeGraphInterface>, std::shared_ptr<rclcpp::node_interfaces::NodeLoggingInterface>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rosidl_action_type_support_t const*, rcl_action_client_options_t const&)::{lambda(rcl_action_client_t*)#1}::operator()(rcl_action_client_t*) const (__closure=0x55ae103eb120, client=0x55ae103c6f00)
    at /root/ros2_ws/colcon_ws/src/ros2/rclcpp/rclcpp_action/src/client.cpp:62
#3  0x00007f5c9e7888c0 in std::_Sp_counted_deleter<rcl_action_client_t*, rclcpp_action::ClientBaseImpl::ClientBaseImpl(std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>, std::shared_ptr<rclcpp::node_interfaces::NodeGraphInterface>, std::shared_ptr<rclcpp::node_interfaces::NodeLoggingInterface>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rosidl_action_type_support_t const*, rcl_action_client_options_t const&)::{lambda(rcl_action_client_t*)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (this=0x55ae103eb110)
    at /usr/include/c++/9/bits/shared_ptr_base.h:471
#4  0x000055ae0ff16130 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x55ae103eb110)
    at /usr/include/c++/9/bits/shared_ptr_base.h:155
#5  0x000055ae0ff12a83 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x55ae103d9b98, 
    __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:730
#6  0x00007f5c9e780990 in std::__shared_ptr<rcl_action_client_t, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (
    this=0x55ae103d9b90, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1169
#7  0x00007f5c9e7809b0 in std::shared_ptr<rcl_action_client_t>::~shared_ptr (this=0x55ae103d9b90, 
    __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr.h:103
#8  0x00007f5c9e7834ea in rclcpp_action::ClientBaseImpl::~ClientBaseImpl (this=0x55ae103d9b40, 
    __in_chrg=<optimized out>) at /root/ros2_ws/colcon_ws/src/ros2/rclcpp/rclcpp_action/src/client.cpp:34
#9  0x00007f5c9e783534 in std::default_delete<rclcpp_action::ClientBaseImpl>::operator() (this=0x55ae103c8b70, 
    __ptr=0x55ae103d9b40) at /usr/include/c++/9/bits/unique_ptr.h:81
#10 0x00007f5c9e7824ca in std::unique_ptr<rclcpp_action::ClientBaseImpl, std::default_delete<rclcpp_action::ClientBaseImpl> >::~unique_ptr (this=0x55ae103c8b70, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:292
#11 0x00007f5c9e77d8fc in rclcpp_action::ClientBase::~ClientBase (this=0x55ae103c8b60, __in_chrg=<optimized out>)
--Type <RET> for more, q to quit, c to continue without paging--
    at /root/ros2_ws/colcon_ws/src/ros2/rclcpp/rclcpp_action/src/client.cpp:137
#12 0x000055ae0ff1d64c in rclcpp_action::Client<tf2_msgs::action::LookupTransform>::~Client (this=0x55ae103c8b60, 
    __in_chrg=<optimized out>) at /root/ros2_ws/colcon_ws/install/rclcpp_action/include/rclcpp_action/client.hpp:622
#13 0x000055ae0ff1d688 in rclcpp_action::Client<tf2_msgs::action::LookupTransform>::~Client (this=0x55ae103c8b60, 
    __in_chrg=<optimized out>) at /root/ros2_ws/colcon_ws/install/rclcpp_action/include/rclcpp_action/client.hpp:622
#14 0x000055ae0ff1bf65 in rclcpp_action::create_client<tf2_msgs::action::LookupTransform>(std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>, std::shared_ptr<rclcpp::node_interfaces::NodeGraphInterface>, std::shared_ptr<rclcpp::node_interfaces::NodeLoggingInterface>, std::shared_ptr<rclcpp::node_interfaces::NodeWaitablesInterface>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<rclcpp::CallbackGroup>, rcl_action_client_options_t const&)::{lambda(rclcpp_action::Client<tf2_msgs::action::LookupTransform>*)#1}::operator()(rclcpp_action::Client<tf2_msgs::action::LookupTransform>*) const (this=0x55ae103ff9f0, ptr=0x55ae103c8b60)
    at /root/ros2_ws/colcon_ws/install/rclcpp_action/include/rclcpp_action/create_client.hpp:79
#15 0x000055ae0ff22d42 in std::_Sp_counted_deleter<rclcpp_action::Client<tf2_msgs::action::LookupTransform>*, rclcpp_action::create_client<tf2_msgs::action::LookupTransform>(std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>, std::shared_ptr<rclcpp::node_interfaces::NodeGraphInterface>, std::shared_ptr<rclcpp::node_interfaces::NodeLoggingInterface>, std::shared_ptr<rclcpp::node_interfaces::NodeWaitablesInterface>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<rclcpp::CallbackGroup>, rcl_action_client_options_t const&)::{lambda(rclcpp_action::Client<tf2_msgs::action::LookupTransform>*)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (this=0x55ae103ff9e0) at /usr/include/c++/9/bits/shared_ptr_base.h:471
#16 0x000055ae0ff16130 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x55ae103ff9e0)
    at /usr/include/c++/9/bits/shared_ptr_base.h:155
#17 0x000055ae0ff12a83 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7f5c6c000c28, 
    __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:730
#18 0x000055ae0ff10d50 in std::__shared_ptr<rclcpp::Waitable, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (
    this=0x7f5c6c000c20, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1169
#19 0x000055ae0ff10d70 in std::shared_ptr<rclcpp::Waitable>::~shared_ptr (this=0x7f5c6c000c20, 
    __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr.h:103
#20 0x00007f5c9e457384 in std::_Destroy<std::shared_ptr<rclcpp::Waitable> > (__pointer=0x7f5c6c000c20)
    at /usr/include/c++/9/bits/stl_construct.h:98
--Type <RET> for more, q to quit, c to continue without paging--
#21 0x00007f5c9e45632d in std::_Destroy_aux<false>::__destroy<std::shared_ptr<rclcpp::Waitable>*> (
    __first=0x7f5c6c000c20, __last=0x7f5c6c000c30) at /usr/include/c++/9/bits/stl_construct.h:108
#22 0x00007f5c9e454e00 in std::_Destroy<std::shared_ptr<rclcpp::Waitable>*> (__first=0x7f5c6c000c20, 
    __last=0x7f5c6c000c30) at /usr/include/c++/9/bits/stl_construct.h:137
#23 0x00007f5c9e4546db in std::_Destroy<std::shared_ptr<rclcpp::Waitable>*, std::shared_ptr<rclcpp::Waitable> > (
    __first=0x7f5c6c000c20, __last=0x7f5c6c000c30) at /usr/include/c++/9/bits/stl_construct.h:206
#24 0x00007f5c9e454049 in std::vector<std::shared_ptr<rclcpp::Waitable>, std::allocator<std::shared_ptr<rclcpp::Waitable> > >::~vector (this=0x55ae10375f80, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:677
#25 0x00007f5c9e4934e2 in rclcpp::memory_strategies::allocator_memory_strategy::AllocatorMemoryStrategy<std::allocator<void> >::~AllocatorMemoryStrategy (this=0x55ae10375f00, __in_chrg=<optimized out>)
    at /root/ros2_ws/colcon_ws/src/ros2/rclcpp/rclcpp/include/rclcpp/strategies/allocator_memory_strategy.hpp:46
#26 0x00007f5c9e4987b9 in __gnu_cxx::new_allocator<rclcpp::memory_strategies::allocator_memory_strategy::AllocatorMemoryStrategy<std::allocator<void> > >::destroy<rclcpp::memory_strategies::allocator_memory_strategy::AllocatorMemoryStrategy<std::allocator<void> > > (this=0x55ae10375f00, __p=0x55ae10375f00) at /usr/include/c++/9/ext/new_allocator.h:153
#27 0x00007f5c9e496ecf in std::allocator_traits<std::allocator<rclcpp::memory_strategies::allocator_memory_strategy::AllocatorMemoryStrategy<std::allocator<void> > > >::destroy<rclcpp::memory_strategies::allocator_memory_strategy::AllocatorMemoryStrategy<std::allocator<void> > > (__a=..., __p=0x55ae10375f00) at /usr/include/c++/9/bits/alloc_traits.h:497
#28 0x00007f5c9e4937cf in std::_Sp_counted_ptr_inplace<rclcpp::memory_strategies::allocator_memory_strategy::AllocatorMemoryStrategy<std::allocator<void> >, std::allocator<rclcpp::memory_strategies::allocator_memory_strategy::AllocatorMemoryStrategy<std::allocator<void> > >, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x55ae10375ef0)
    at /usr/include/c++/9/bits/shared_ptr_base.h:557
#29 0x000055ae0ff16130 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x55ae10375ef0)
    at /usr/include/c++/9/bits/shared_ptr_base.h:155
#30 0x000055ae0ff12a83 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x55ae10375cf0, 
    __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:730
#31 0x000055ae0ff10848 in std::__shared_ptr<rclcpp::memory_strategy::MemoryStrategy, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x55ae10375ce8, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1169
#32 0x000055ae0ff10868 in std::shared_ptr<rclcpp::memory_strategy::MemoryStrategy>::~shared_ptr (this=0x55ae10375ce8, 
    __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr.h:103
--Type <RET> for more, q to quit, c to continue without paging--
#33 0x00007f5c9e45a41a in rclcpp::Executor::~Executor (this=0x55ae10375c28, __in_chrg=<optimized out>)
    at /root/ros2_ws/colcon_ws/src/ros2/rclcpp/rclcpp/src/rclcpp/executor.cpp:95
#34 0x00007f5c9e4711b2 in rclcpp::executors::SingleThreadedExecutor::~SingleThreadedExecutor (this=0x55ae10375c28, 
    __in_chrg=<optimized out>)
    at /root/ros2_ws/colcon_ws/src/ros2/rclcpp/rclcpp/src/rclcpp/executors/single_threaded_executor.cpp:24
#35 0x000055ae0ff127a6 in TestBufferClient::~TestBufferClient (this=0x55ae10375bf0, __in_chrg=<optimized out>)
    at /root/ros2_ws/colcon_ws/src/ros2/geometry2/tf2_ros/test/test_buffer_client.cpp:103
#36 0x000055ae0ff22af8 in TestBufferClient_lookup_transform_available_Test::~TestBufferClient_lookup_transform_available_Test (this=0x55ae10375bf0, __in_chrg=<optimized out>)
    at /root/ros2_ws/colcon_ws/src/ros2/geometry2/tf2_ros/test/test_buffer_client.cpp:156
#37 0x000055ae0ff22b18 in TestBufferClient_lookup_transform_available_Test::~TestBufferClient_lookup_transform_available_Test (this=0x55ae10375bf0, __in_chrg=<optimized out>)
    at /root/ros2_ws/colcon_ws/src/ros2/geometry2/tf2_ros/test/test_buffer_client.cpp:156
#38 0x000055ae0ff53cd2 in testing::Test::DeleteSelf_ (this=0x55ae10375bf0)
    at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:492
#39 0x000055ae0ff63e5a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (
    object=0x55ae10375bf0, 
    method=(void (testing::Test::*)(class testing::Test * const)) 0x55ae0ff53ca4 <testing::Test::DeleteSelf_()>, 
    location=0x55ae0ff7c267 "the test fixture's destructor")
    at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2433
#40 0x000055ae0ff5c8e5 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (
    object=0x55ae10375bf0, 
    method=(void (testing::Test::*)(class testing::Test * const)) 0x55ae0ff53ca4 <testing::Test::DeleteSelf_()>, 
    location=0x55ae0ff7c267 "the test fixture's destructor")
    at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2469
#41 0x000055ae0ff37f62 in testing::TestInfo::Run (this=0x55ae103704b0)
    at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2690
#42 0x000055ae0ff38656 in testing::TestSuite::Run (this=0x55ae103709c0)
    at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2816
--Type <RET> for more, q to quit, c to continue without paging--
#43 0x000055ae0ff4473c in testing::internal::UnitTestImpl::RunAllTests (this=0x55ae10370680)
    at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:5338
#44 0x000055ae0ff65388 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x55ae10370680, 
    method=(bool (testing::internal::UnitTestImpl::*)(class testing::internal::UnitTestImpl * const)) 0x55ae0ff44334 <testing::internal::UnitTestImpl::RunAllTests()>, 
    location=0x55ae0ff7cb18 "auxiliary test code (environments or event listeners)")
    at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2433
#45 0x000055ae0ff5db23 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x55ae10370680, 
    method=(bool (testing::internal::UnitTestImpl::*)(class testing::internal::UnitTestImpl * const)) 0x55ae0ff44334 <testing::internal::UnitTestImpl::RunAllTests()>, 
    location=0x55ae0ff7cb18 "auxiliary test code (environments or event listeners)")
    at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2469
#46 0x000055ae0ff42f58 in testing::UnitTest::Run (this=0x55ae0ffb64c0 <testing::UnitTest::GetInstance()::instance>)
    at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:4925
#47 0x000055ae0ff0ff04 in RUN_ALL_TESTS ()
    at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:2473
#48 0x000055ae0ff0f28e in main (argc=1, argv=0x7ffd561349f8)
    at /root/ros2_ws/colcon_ws/src/ros2/geometry2/tf2_ros/test/test_buffer_client.cpp:197
``` 
</div></details>

@zhoulaifu
Copy link
Author

zhoulaifu commented Feb 15, 2021

Just find out that the same memory leaks can be exposed by running three other binaries (besides tf2_ros/test_buffer_client) from geometry2, with Address Sanitizers on:

  • test_tf2/test_message_filter
  • tf2_ros/buffer_server
  • tf2_ros/tf2_ros_test_time_reset

The leaks can be reproduced on the ros:foxy docker with the same dockerfile above. I have not tested with the recent rclcpp though (due to difficulties with source compilation).

@zhoulaifu
Copy link
Author

Besides, the same memory leak can also be triggered without sanitizers. Just compile with plain colcon build and run tf2_ros/test_buffer_client a couple of times triggers the bug on my machine.

for i in {1..20}; do ${ROS_WS}/build/tf2_ros/test_buffer_client ;done

@zhoulaifu
Copy link
Author

zhoulaifu commented Feb 21, 2021

Sorry. Just realized that the memory leaks have been addressed recently.

@fujitatomoya
Copy link
Collaborator

we've fixed some memory leaks in a couple of month, but i can still see this problem.

@fujitatomoya
Copy link
Collaborator

@zhoulaifu

could you try #1562? i tried more than hundreds times, no memory leak observed.

@zhoulaifu
Copy link
Author

Just saw this. @fujitatomoya you mean try the newly merged one (related to another issue) ? By the way, I just realize that no Continuous Integration is used for rclcpp: https://index.ros.org/r/rclcpp/, or I may misunderstand?

@clalancette
Copy link
Contributor

By the way, I just realize that no Continuous Integration is used for rclcpp: https://index.ros.org/r/rclcpp/, or I may misunderstand?

Unfortunately, the index doesn't understand all of the ways we do Continuous Integration. We actually have several forms of CI for rclcpp (and all of the core ROS 2 packages, for that matter).

Every time a PR is opened a "PR job" is run (you can see one of the runs for rclcpp here). That job is helpful to give early feedback. The downside is that it only runs on Linux, and cannot test API changes across repositories.

To handle the latter two things, we always run jobs on https://ci.ros2.org on all platforms, with all core packages built from source. You can see what that looks like here: #1562 (comment)

@fujitatomoya
Copy link
Collaborator

@zhoulaifu

you mean try the newly merged one

yes, you can use either rolling or foxy from the source. (I already did confirm the problem is gone, but just in case) or if it is okay for you to close issue, we can do that too.

@zhoulaifu
Copy link
Author

@fujitatomoya Could you share a Dockerfile or something that specifies your environment where this bug disappears? Probably i misuse something as I still got the error with the following Dockerfile.


FROM ros:rolling

ENV ROS_WS /opt/ros_ws
WORKDIR $ROS_WS

RUN cd $ROS_WS \
        &&  git clone -b foxy --depth 1 https://github.com/ros2/geometry2 $ROS_WS/src/geometry2 \
        && . /opt/ros/${ROS_DISTRO}/setup.sh \
        &&  rm -rf ${ROS_WS}/build && VERBOSE=1 CXX="g++" CXXFLAGS="--coverage -g -fsanitize=address" colcon build --event-handlers console_direct+

Perhaps something is wrong with this Dockerfile since it mixes rolling and foxy?

@clalancette
Copy link
Contributor

Perhaps something is wrong with this Dockerfile since it mixes rolling and foxy?

Yes, that's a problem (we haven't backported all of the fixes from rolling to foxy). You should change that to:

RUN cd $ROS_WS \
        &&  git clone -b ros2 --depth 1 https://github.com/ros2/geometry2 $ROS_WS/src/geometry2 \
        && . /opt/ros/${ROS_DISTRO}/setup.sh \
        &&  rm -rf ${ROS_WS}/build && VERBOSE=1 CXX="g++" CXXFLAGS="--coverage -g -fsanitize=address" colcon build --event-handlers console_direct+

(notice that we checkout the ros2 branch now)

@zhoulaifu
Copy link
Author

@fujitatomoya FYI, I tried with a corrected dockerfile (thanks to @clalancette),

FROM ros:rolling

ENV ROS_WS /opt/ros_ws
WORKDIR $ROS_WS

RUN cd $ROS_WS \
        &&  git clone -b ros2 --depth 1 https://github.com/ros2/geometry2 $ROS_WS/src/geometry2 \
        && . /opt/ros/${ROS_DISTRO}/setup.sh \
        &&  rm -rf ${ROS_WS}/build && VERBOSE=1 CXX="g++" CXXFLAGS="--coverage -g -fsanitize=address" colcon build --event-handlers console_direct+


I got no errors with a single run, but still got memory leaks after many runs

  • This is okay: ./build/tf2_ros/test_buffer_client
  • This triggers memory leaks again: for i in {1..10}; do ./build/tf2_ros/test_buffer_client; done

@clalancette
Copy link
Contributor

Can you give us the stack trace of the leak?

@fujitatomoya
Copy link
Collaborator

@zhoulaifu

i tried with source build https://github.com/ros2/ros2/blob/master/ros2.repos, cannot make that happen.

@zhoulaifu
Copy link
Author

Okay @fujitatomoya Then please ignore my message above. I thought you meant that testing on rolling would be, in this case, the same.

Also, I do not know if this is easily doable: is it possible to have a Dockerfile published on the ROS website which specifies the build instructions of ROS2 from the latest source. In this way, if one finds a bug in a release version such as foxy, he/she can readily check it again on your latest source. For now, building ROS from source following instructions from the ROS website remains a tedious task for amateurs unfortunately, which could slow down the process of bug finding/reporting/confirming.

@fujitatomoya
Copy link
Collaborator

@zhoulaifu thanks for the consideration, please feel free to re-open or open new issue anytime 😃

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working more-information-needed Further information is required
Projects
None yet
Development

No branches or pull requests

3 participants