Launch-based Testing

[hidmic]

Feature request

This is a feature request to enable launch-based testing, using (and likely extending) the current launch API. Acceptance criteria is limited to running both gtest and stdout-matching launch tests through regular build tools (e.g. colcon).

Feature description

In ROS, the rostest integraton with roslaunch allows for running test nodes in otherwise regular launch files.
When run with the rostest CLI, these nodes (either gtest or nosetests based executables) make the whole launch test fail (or succeed) based on their return code. Fixture nodes can also cause a test failure by means of the required tag. All test nodes have user specified timeouts. Integration with catkin (i.e. CMake) is readily available too.

Similarly, though in greater generality, launch_testing should provide the means to:

• Describe an executable under test, along with an executable or a group thereof to act as test fixture.
• Assert outcome of the executable under test (return code, stdout, stderr, etc.), which may
  or may not be using a testing framework itself (e.g. gtest, pytest, etc.).
• Assert a failure after user specified test timeout.
• Optionally assert a failure upon normal or abnormal termination of test fixture executables.
• Teardown the whole running launch test description upon assertion failure.
• Aggregate test outcome in a standard file format for latter recollection (e.g. xunit files).
• Integrate launch test descriptions as yet another package test instance, using specific CMake API.

Implementation considerations

launch_testing

To maximize code reuse, test descriptions must compose executable descriptions (see
#114 for a precedent on this pattern).

For test descriptions, assertions can be coupled or decoupled from their respective conditions. Compare:

GTest(Node(package='demos_rclcpp', node_executable='talker')),
MatchingTest(Node(package='demos_rclpy', node_executable='listener'), matching_pattern='.*ERROR.*')

where each new test kind is a subclass of a base Test action (an Action subclass itself), with

Assert(GTestPasses(Node())),
Expect(OutputMatches(Node()))

where two Action subclasses, Assert (terminating) and Expect (non-terminating), take a Condition instance that in turn deals with the executables.

In either case, the composing objects will setup timers to timeout and process event handlers to collect output, deal with testing framework details if need be (e.g. argument passing) and emit specific events (i.e. AssertionError) upon test failure. Then, a tests-aware launch.LaunchService may react accordingly, shutting down the whole launch description and aggregating tests data.

Note that in (1), defining launch description behavior upon executable termination becomes necessary (orthogonal to testing, but instrumental to mimic'ing ROS required tag usage for fixture executables). See the legacy exit handlers implementation for feature reference (though implementation in this case may rely on events).

ros2launch

CLI must be updated with test verb to easily invoke a test-aware launch.LaunchService.

[hidmic]

On a first step, we will:

1. Write some testing examples on how the API would look like. (https://github.com/ros2/examples/issues/223).
2. Put together API basic functionality (no result aggregation, single test per launch, etc.),
   migrating legacy features if needed. (Add actions for running gtest and pytests #176)
3. Add basic CMake integration. ([launch] Add basic CMake support to run launch-based tests #162)
4. Migrate existing legacy launch API use cases. (Remove legacy API #159)

[hidmic]

Basic testing functionality is provided by the launch_testing package after #167. A feature complete equivalent of ROS 1 rostest is on the making at ApexAI/apex_rostest.

[hidmic]

Functionality provided here has been superseded by that introduced on #208. Refer to that issue for further reference.