Add pub/sub init, publish and take instrumentation using tracetools

[christophebedard]

This adds tracing instrumentation using tracetools for publisher/subscription initialization at the rmw level. By recording the pub/sub GIDs, we can match ROS 2 pubs/subs with the corresponding DDS writers/readers.

This also adds a tracepoint for rmw_publish and rmw_take so that we can track published/taken messages at the rmw level.

For simplicity, this only adds instrumentation for rmw_fastrtps_cpp and not rmw_fastrtps_dynamic_cpp, since the former is the default RMW implementation while the latter is a tier 2 implementation.

Equivalent to this PR for rmw_cyclonedds_cpp: ros2/rmw_cyclonedds#329. Since the default RMW implementation was changed from rmw_cyclonedds_cpp to rmw_fastrtps_cpp, it would be great to support rmw_fastrtps_cpp.

Signed-off-by: Christophe Bedard bedard.christophe@gmail.com

[christophebedard]

CI testing rmw_fastrtps_cpp rmw_fastrtps_shared_cpp rmw_implementation test_rmw_implementation:

• Linux
• Linux-aarch64
• Windows

[WideAwakeTN]

Is the TRACEPOINT macro dangerous for hard realtime (i.e. using a mutex or the system heap)? The use of TRACEPOINT in rmw_publish.cpp and rmw_take.cpp could then be problematic.

[christophebedard]

Is the TRACEPOINT macro dangerous for hard realtime (i.e. using a mutex or the system heap)? The use of TRACEPOINT in rmw_publish.cpp and rmw_take.cpp could then be problematic.

I think this is a pretty simplistic question; there is a bit more to it:

There are a bunch of TRACEPOINTs here and there in ROS 2, not just here. This macro itself is just a function call, and by default, unless you install the LTTng tracer and re-build ROS 2, the function will be empty.

I do think it would be worth including the LTTng tracepoints by default (see ros2/ros2#1177). In this case or if you install LTTng and re-build ROS 2, if you do not setup LTTng + enable that tracepoint event, it only does a simple check that has virtually no impact at all. If you do enable the event, it can be compatible with real-time if you set up LTTng correctly, see: https://gitlab.com/ros-tracing/ros2_tracing#real-time.

If you really have a real-time application (hard real-time or not), chances are you don't use ROS 2 as-is; you probably have a slightly-modified version that you build yourself, so you can just enable the right build option and remove all TRACEPOINT() calls if you want: https://gitlab.com/ros-tracing/ros2_tracing#disabling-tracing

[fujitatomoya]

@WideAwakeTN

Is the TRACEPOINT macro dangerous for hard realtime

what is your definition about hard realtime?

(i.e. using a mutex or the system heap)?

btw, these are used everywhere in ROS 2 system.

i think hardware realtime system is, (probably my definition, maybe not correct...)

• it must absolutely hit every deadline.
• it must be able to reproduce the same problem(deadline miss).

if this is what your are expecting for ROS 2 system as-is, probably ROS 2 cannot cover your requirements?

[WideAwakeTN]

There are a bunch of TRACEPOINTs here and there in ROS 2, not just here. This macro itself is just a function call, and by default, unless you install the LTTng tracer and re-build ROS 2, the function will be empty.

That sound's fair enough.

what is your definition about hard realtime

I think it's similar to yours. In the context of DDS experienced that people often use the term realtime in a soft or loose manner, i.e. expressing that code is "fast". I use the term hard realtime to distinguish between "fast" code and code that is not interrupted unexpectedly with timing characteristics similar to that of a microcontroller (deadlines, repeatability).

probably ROS 2 cannot cover your requirements

My impression was the goal of ROS 2 is to be hard realtime. Having a single system heap access in a cyclic phase destroys this goal, which is what I have witnessed personally in a project in which I was involved. This made me sensitive to this issue. Heap access in a realtime critical code creates an interaction with all other heap accessing threads in a system, which leads to total unpredictability and hard to dignose errors at the customer. In general, lock free code is not so easy to achieve.

Is there some documentation available on what is holding Ros2 back from achieving hard realtime at the moment? Depending on how my job life evolves I may be getting more involved with Ros2 development in the future. I also don't want to reinvent the wheel, I saw that happening over and over too.

Thanks for getting me informed.

[fujitatomoya]

Is there some documentation available on what is holding Ros2 back from achieving hard realtime at the moment? Depending on how my job life evolves I may be getting more involved with Ros2 development in the future. I also don't want to reinvent the wheel, I saw that happening over and over too.

one thing that i can introduce is we have https://github.com/ros-realtime/community. that could be the place we can discuss details and plans, i think.

[fujitatomoya]

@christophebedard can you fix the conflict? this needs to be merged in today.

[christophebedard]

@fujitatomoya I rebased to fix the conflicts. New CI run just to be safe:

• Linux
• Linux-aarch64
• Windows

[christophebedard]

I think the warning on Windows is from #513

rmw_fastrtps/rmw_fastrtps_shared_cpp/src/TypeSupport_impl.cpp

Line 446 in 490f06e

uint32_t bound = member->string_upper_bound_ ? member->string_upper_bound_ : 255;

[fujitatomoya]

@christophebedard right, this warning is unrelated to this PR.