Interop with DDS application using existing .idl files

[maspe36]

I'm trying to create a ROS2 node that can listen to messages being published from an existing DDS application. I think I've checked most of the regular interop checkboxes, however still no luck with the interop

• Same domain id
• Same QoS settings
• Both using RTI DDS
• Topic name starts with rt/
• Type is under two modules which line up with my ros2 package

I created a message package where I try to pass my existing .idl files into the rosidl_generate_interfaces cmake function. However it seems like we depend on rosidl_typesupport_fastrtps_cpp to parse the idl files. This means I cannot use the exact same .idl files as I do in my existing DDS application. Even when I strip the .idl down (No constants, no usings, etc..) to something that is parseable by fastrtps, the generated type doesn't seem to lineup with the type used by my DDS application. Verified this assumption in rtiadminconsole

.idl's as they are defined in my ROS2 package

my_msgs/msg/header.idl

module my_msgs {
    module msg {
        struct Header {
            long id;  //@key
            unsigned long long sendCount;
        };
    };
};

my_msgs/msg/telemetry.idl

#include "header.idl"

module my_msgs {
    module msg {
        struct Telemetry {
            my_msgs::msg::Header header;  //@key
            sequence<octet, 65536> data;
        };
    };
};

CMakeLists.txt

cmake_minimum_required(VERSION 3.8)
project(my_msgs)

if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
  add_compile_options(-Wall -Wextra -Wpedantic)
endif()

# find dependencies
find_package(ament_cmake REQUIRED)

find_package(rosidl_default_generators REQUIRED)

rosidl_generate_interfaces(${PROJECT_NAME}
        "msg/header.idl"
        "msg/telemetry.idl"
        )

if(BUILD_TESTING)
  find_package(ament_lint_auto REQUIRED)
  # the following line skips the linter which checks for copyrights
  # uncomment the line when a copyright and license is not present in all source files
  #set(ament_cmake_copyright_FOUND TRUE)
  # the following line skips cpplint (only works in a git repo)
  # uncomment the line when this package is not in a git repo
  #set(ament_cmake_cpplint_FOUND TRUE)
  ament_lint_auto_find_test_dependencies()
endif()

ament_package()

package.xml

<?xml version="1.0"?>
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
  <name>my_msgs</name>
  <version>0.0.0</version>
  <description>TODO: Package description</description>
  <maintainer email="<real email>">sam</maintainer>
  <license>TODO: License declaration</license>

  <buildtool_depend>ament_cmake</buildtool_depend>

  <build_depend>rosidl_default_generators</build_depend>
  <exec_depend>rosidl_default_runtime</exec_depend>
  <member_of_group>rosidl_interface_packages</member_of_group>

  <test_depend>ament_lint_auto</test_depend>
  <test_depend>ament_lint_common</test_depend>

  <export>
    <build_type>ament_cmake</build_type>
  </export>
</package>

Defined in my DDS application

header.idl

module my_msgs {
    module msg {
        struct Header {
            long id;  //@key
            unsigned long long sendCount;
        };
    };
};

telemetry.idl

#include "header.idl"

module my_msgs {
    module msg {
        struct Telemetry {
            Header header;  //@key
            sequence<octet, 65536> data;
        };
    };
};

It seems like I should be using rosidl_typesupport_connext as it uses rtiddsgen. But its not supported for galactic and up under the guise that this is the new rmw implementation ros2/rosidl_typesupport_connext#70.

Am I missing something?

[asorbini]

Hi @maspe36,

The problem comes from your use of keys, which is not supported by ROS 2. Even though the ROS 2 code generator is able to parse your IDL, I suspect that the // @key annotation is ignored. In fact, the preferred ways to define types in ROS 2 (i.e. via .msg, .srv, and .action files) do not provide any syntax to mark members as keys.

However it seems like we depend on rosidl_typesupport_fastrtps_cpp to parse the idl files. This means I cannot use the exact same .idl files as I do in my existing DDS application.

rmw_connextdds re-uses the "type support" code generated by rosidl_typesupport_fastrtps_cpp in order to allow Connext to be able to serialize/deserialize ROS 2 messages into the "wire representation" (XCDR) used to share them over the network.

It seems like I should be using rosidl_typesupport_connext as it uses rtiddsgen. But its not supported for galactic and up under the guise that this is the new rmw implementation ros2/rosidl_typesupport_connext#70.

The solution implemented by rosidl_typesupport_connext which was used by the previous Connext RMW (rmw_connext_cpp) had a fundamental design flaw which forced an extra copy to occur every time data was passed between Connext and ROS 2. Additionally, even though it relied on code generation with rtiddsgen, you still wouldn't have been able to use keyed types with it, since they weren't supported either. As I said, the problem is that even though ROS 2 is built on top of DDS, it does not support keys in its data model.

Now, coming to possible solutions to your problem, there aren't any available if you need to use keys and want to rely only on the ROS 2 API.

If you want to continue to use keys, you could extend your ROS 2 application to access the underlying DDS DomainParticipant using Connext's "modern" C++ API, assuming you are running with rmw_connextdds. You will then be able to create Connext DataReaders and DataWriters as you would normally do in any Connext application. This is the easiest way to access all features provided by Connext in your ROS 2 application. The main drawback of this approach is that the events coming from Connext will have to be handled separately from the ROS 2 "executor" loop, potentially introducing some synchronization needs at the application level.

I have personally been experimenting with creating some helper APIs to simplify this use case of "hybrid" ROS 2/Connext applications by extending the Node class with some helper methods to create readers and writers in a more ROS-like fashion. The APIs also automatically apply ROS naming conventions for the topic name and support resolving topic name remappings.

If using Connext's native APIs is not an option for some reason, you will need to further modify the data model used by your ROS 2 components to drop the key annotations. This of course will also mean changing the data types on your "native" Connext endpoints in order to make them match, and, maybe most importantly, losing all the advantages provided by topic instances. If you want to go down this road, and try to limit the scope of "data model changes" in your system, you might want to consider introducing a "bridge component" to mediate between the types, possibly implemented by using the SDK provided by RTI Routing Service.

[maspe36]

Hey thanks for the detailed response! I will say upfront I am no expert but here goes nothing.

I suspect that the // @key annotation is ignored. In fact, the preferred ways to define types in ROS 2 (i.e. via .msg, .srv, and .action files) do not provide any syntax to mark members as keys.

Yeah I did some reading on this before I started working on this and I stumbled across a couple, of discussions that seem to imply that it is up to the rmw implementation to use the keyed field. The generated "rosidl" file actually does have the //@key annotation still, which as I understood it would be passed to the rmw implementation.

rmw_connextdds re-uses the "type support" code generated by rosidl_typesupport_fastrtps_cpp in order to allow Connext to be able to serialize/deserialize ROS 2 messages into the "wire representation" (XCDR) used to share them over the network.

I can understand re-using code, but in this case you can't even parse the same idl files as a native application. For example in the top post, you can see I had to explicitly add the module namespace to my included type (my_msgs::msg::Header vs Header) despite being in the same module. It would appear to me that is also a cause of the Type Consistency error I'm seeing in the admin console.

had a fundamental design flaw which forced an extra copy to occur every time data was passed between Connext and ROS 2.

Was this caused by rtiddsgen? Seems to be something external to idl parsing during a build?

experimenting with creating some helper APIs to simplify this use case of "hybrid" ROS 2/Connext applications by extending the Node class with some helper methods to create readers and writers in a more ROS-like fashion

Wow this looks great. This is exactly what my mind jumped to as I was reading your top comments. I'll take a look at this and maybe experiment myself with a python version.

[superware]

Hello @maspe36 @asorbini I'm also trying to utilize RTI Connext @key, is there any workaround in mind, which might involve patching ROS2 or rmw_connextdds, that will allow defining a field as key? (even without ROS2 native support)

Thanks!