From 3537d6abeb48e3372451521cb04b989229d99616 Mon Sep 17 00:00:00 2001 From: Arne Baeyens Date: Sun, 1 Dec 2024 11:27:20 +0100 Subject: [PATCH 01/15] tutorial/integration_testing: first draft Signed-off-by: Arne Baeyens --- .../Intermediate/Testing/Integration.rst | 432 ++++++++++++++++++ .../Intermediate/Testing/Testing-Main.rst | 1 + 2 files changed, 433 insertions(+) create mode 100644 source/Tutorials/Intermediate/Testing/Integration.rst diff --git a/source/Tutorials/Intermediate/Testing/Integration.rst b/source/Tutorials/Intermediate/Testing/Integration.rst new file mode 100644 index 00000000000..eafd854dd30 --- /dev/null +++ b/source/Tutorials/Intermediate/Testing/Integration.rst @@ -0,0 +1,432 @@ +.. TestingIntegration: + +Writing Basic Integration Tests with launch_testing +=================================================== + +**Goal:** Create and run integration tests on the ROS 2 Turtlesim node. + +**Tutorial level:** Intermediate + +**Time:** 20 minutes + +.. contents:: Contents + :depth: 2 + :local: + + +Prerequisites +------------- +Before starting this tutorial, +it is recommended to have completed the following tutorials on launching nodes: + +* :doc:`Launching Multiple Nodes <../../Beginner-CLI-Tools/Launching-Multiple-Nodes/Launching-Multiple-Nodes>` +* :doc:`Creating Launch files <../../Intermediate/Launch/Creating-Launch-Files>` + + +Background +---------- +Where unit tests focus on validating a very specific piece of functionality, +integration tests go all the way in the other direction. +In ROS 2 this means launching a system of one or several nodes, +for example the Gazebo simulator and the Nav2 navigation stack. +As a result, these tests are more complex both to set up and to run. + +A key aspect of ROS 2 integration testing is that nodes part of different tests +shouldn't communicate with each other, even when run in parallel, +which will be achieved here using a specific test runner +that picks unique ROS domain id's. +In addition, integration tests have to fit in the overall testing workflow. +A standardized approach is to ensure each test outputs an XUnit file, +which are easily parsed using common test tooling. + + +Overview +-------- +The main tool in use here is the +`launch_testing `_ +package +(`launch_testing repository `_). +This ROS-agnostic functionality allows to extend a classic Python launch file +with both active tests (that run while the nodes are also running) +and post-shutdown tests (which run once after all nodes have exited). +``launch_testing`` relies on the Python standard module +`unittest `_ +for the actual testing. +Hence, preferably avoid mixing with ``pytest``. +If you prefer using pytest for integration tests, +you will want to use the package +`launch_pytest `_ +(`launch_pytest repository `_), +though this tutorial will discuss using unittest. + +To get our integration tests run as part of ``colcon test``, +we register the launch file in the ``CMakeLists.txt``. +To finish, we will briefly touch +on how to run these tests and inspect the test results. + + +Steps +----- + +1 Describe the test in the test launch file +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Both the nodes under test and the tests themselves are launched +using a Python launch file, which resembles a classic ROS 2 Python launch file. +It is custom to let the integration test launch file names +follow the pattern ``test/test_*.py``. + +There are two common types of tests in integration testing: +active tests, which run while the nodes under test are running, +and post-shutdown tests, which are run after exiting the nodes. +We will cover both in this tutorial. + + +1.1 Imports +~~~~~~~~~~~ +The top imports the Python modules we will be using. +Only two modules are specific to testing: +the general-purpose ``unittest``, and ``launch_testing``. + +.. code-block:: python + + import os + import sys + import time + import unittest + + import launch + import launch_ros + import launch_testing.actions + import rclpy + from turtlesim.msg import Pose + + +1.2 Generate the test description +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The function ``generate_test_description`` describes what to launch, +similar to ``generate_launch_description`` +in a classic ROS 2 Python launch file. +In the example below, we launch the turtlesim node +and half a second later our tests. + +In more complex integration test setups, you will probably want +to launch a system of several nodes, together with additional nodes +that performing mocking or must otherwise interact with the nodes under test. +Including existing (XML) launch files - to avoid launch file duplication - +is supported. + +.. code-block:: python + + def generate_test_description(): + return ( + launch.LaunchDescription( + [ + # Nodes under test + launch_ros.actions.Node( + package='turtlesim', + namespace='', + executable='turtlesim_node', + name='turtle1', + ), + # Launch tests 0.5 s later + launch.actions.TimerAction( + period=0.5, actions=[launch_testing.actions.ReadyToTest()]), + ] + ), {}, + ) + + +1.3 Active tests +~~~~~~~~~~~~~~~~ +The active tests interact with the running nodes. In this tutorial, +we will check whether the Turtlesim node publishes pose messages +(by listening to the node's 'turtle1/pose' topic) +and whether it logs that it spawned the turtle +(by listening to stderr). + +The active tests are defined as methods of a class inheriting +from `unittest.TestCase `_. +The child class, here ``TestTurtleSim``, contains the following methods: + +- ``test_*``: + the test methods, each performing some ROS communication + with the nodes under test and/or listening to the process output + (passed in through ``proc_output``). + They are executed sequentially. +- ``setUp``, ``tearDown``: + respectively run before (to prepare the test fixture) + and after executing each test method. + By creating the node in the ``setUp`` method, + we use a different node instance for each test + to reduce the risk of tests contaminating each other. +- ``setUpClass``, ``tearDownClass``: + these class methods respectively run once before and after + executing all the test methods. + +It's highly recommended to go through +`launch_testing's detailed documentation on this topic `_. + +.. code-block:: python + + # Active tests + class TestTurtleSim(unittest.TestCase): + @classmethod + def setUpClass(cls): + rclpy.init() + + @classmethod + def tearDownClass(cls): + rclpy.shutdown() + + def setUp(self): + self.node = rclpy.create_node('test_turtlesim') + + def tearDown(self): + self.node.destroy_node() + + def test_publishes_pose(self, proc_output): + """Check whether pose messages published""" + msgs_rx = [] + sub = self.node.create_subscription( + Pose, 'turtle1/pose', + lambda msg: msgs_rx.append(msg), 100) + try: + # Listen to the pose topic for 10 s + end_time = time.time() + 10 + while time.time() < end_time: + # spin to get subscriber callback executed + rclpy.spin_once(self.node, timeout_sec=1) + # There should have been 100 messages received + assert len(msgs_rx) > 100 + finally: + self.node.destroy_subscription(sub) + + def test_logs_spawning(self, proc_output): + """Check whether logging properly""" + proc_output.assertWaitFor( + 'Spawning turtle [turtle1] at x=', + timeout=5, stream='stderr') + +Note that the way we listen to the 'turtle1/pose' topic +in ``test_publishes_pose`` differs from +:doc:`the usual approach <../../Beginner-Client-Libraries/Writing-A-Simple-Py-Publisher-And-Subscriber>`. +Instead of calling the blocking ``rclpy.spin``, +we trigger the ``spin_once`` method - +which executes the first open callback +(so, our subscriber callback if a message arrived within 1 s) - +until we have gathered all messages published over the last 10 s. + +If you want to go further, you can implement yourself a third test +that publishes a twist message, asking the turtle to move, +and subsequently checks that it moved +by asserting that the pose message changed, +effectively automating part of the +`Turtlesim introduction tutorial <../../Beginner-CLI-Tools/Introducing-Turtlesim/Introducing-Turtlesim>`. + + +1.4 Post-shutdown tests +~~~~~~~~~~~~~~~~~~~~~~~ +The classes marked with the ``launch_testing.post_shutdown_test`` decorator +are run after letting the nodes under test exit. +A typical test here is whether the nodes exited cleanly, +for which ``launch_testing`` provides the method +`asserts.assertExitCodes `_. + +.. code-block:: python + + # Post-shutdown tests + @launch_testing.post_shutdown_test() + class TestTurtleSimShutdown(unittest.TestCase): + def test_exit_codes(self, proc_info): + """Check if the processes exited normally.""" + launch_testing.asserts.assertExitCodes(proc_info) + + +2 Register the test in the CMakeLists.txt +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Registering the test in the ``CMakeLists.txt`` fulfills two functions: +it integrates it in the ``CTest`` framework ROS 2 CMake-based packages rely on +(and hence it will be called when running ``colcon test``), +and it also allows to specify *how* the test is to be run - +in this case, with a unique domain id to ensure test isolation. +This latter aspect is realized using the special test runner +`run_test_isolated.py `_. +To ease adding several integration tests, +we define the CMake function ``add_ros_isolated_launch_test`` +such that each additional test requires only a single line. + +.. code-block:: cmake + + cmake_minimum_required(VERSION 3.8) + project(app) + + ######## + # test # + ######## + + if(BUILD_TESTING) + # Integration tests + find_package(ament_cmake_ros REQUIRED) + find_package(launch_testing_ament_cmake REQUIRED) + function(add_ros_isolated_launch_test path) + set(RUNNER "${ament_cmake_ros_DIR}/run_test_isolated.py") + add_launch_test("${path}" RUNNER "${RUNNER}" ${ARGN}) + endfunction() + add_ros_isolated_launch_test(test/test_integration.py) + endif() + + +3 Dependencies and package organization +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Finally, to avoid surprises, +add the following dependencies to your ``package.xml``: + +.. code-block:: XML + + ament_cmake_ros + launch + launch_ros + launch_testing + launch_testing_ament_cmake + rclpy + turtlesim + + +After following the above steps, your package (here named 'app') +ought to look as follows: + +.. code-block:: + + app/ + CMakeLists.txt + package.xml + tests/ + test_integration.py + +Concerning package organization: +Integration tests can be part of any ROS package. +One can dedicate one or more packages to just integration testing, +or alternatively add them to the package of which they test the functionality. +In this tutorial, we go with the first option +as we will test the existing Turtlesim node. + + +4 Running tests and report generation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +4.1 Running with colcon +~~~~~~~~~~~~~~~~~~~~~~~ +Running all tests is straightforward: simply run +:doc:`colcon test <../../Intermediate/Testing/CLI>`. +This command suppresses the test output +and exposes little about which tests succeed and which fail. +Useful therefore while developing tests is the option +to print all test output while the tests are running: + +.. code-block:: console + + colcon test --event-handlers console_direct+ + + +4.2 Visualizing test results +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +For viewing the results, there's a separate colcon verb. For example, + +.. code-block:: console + + $ colcon test-result --all + build/app/Testing/20241013-0810/Test.xml: 1 tests, 0 errors, 1 failure, 0 skipped + build/app/test_results/app/test_test_integration.py.xunit.xml: 3 tests, 0 errors, 1 failure, 0 skipped + + Summary: 4 tests, 0 errors, 2 failures, 0 skipped + +lists two files: +one ctest-formatted XML file (a result of the ``CMakeLists.txt``) and, +more interestingly, also an XUnit-formatted XML file. +This latter is suitable for automatic report generation +in automated testing in CI/CD pipelines. +If we would have also added unit tests, their XUnit files +would show up as well here. + +A suitable tool to visualize them all together is the +`NodeJS package Xunit Viewer `_. +It converts the XUnit files to HTML or straight into the terminal. +For example, command and response (without highlighting): + +.. code-block:: console + + $ xunit-viewer -r build/app/test_results -c + app.test_integration.launch_tests + ✗ test_publishes_pose time=0.52 + - Traceback (most recent call last): + File "/home/user/ros_workspace/src/app/test/test_integration.py", line 67, in test_publishes_pose + assert len(msgs_rx) > 100 + ^^^^^^^^^^^^^^^^^^ + AssertionError + ✓ test_exit_codes time=0.0 + ✓ test_logs_spawning time=0.197 + + 1 failure, 2 passed + Written to: /home/user/ros_workspace/index.html + + +Summary +------- + +In this tutorial, we explored the process of creating and running +integration tests on the ROS 2 Turtlesim node. +We discussed the integration test launch file +and covered writing active tests and post-shutdown tests. +To recap, the four key elements of the integration test launch file are: + +* The function ``generate_test_description``: + same as the classic way of launching nodes + (basically, it replaces ``generate_launch_description``). + It launches our nodes under tests as well as our tests. +* ``launch_testing.actions.ReadyToTest()``: + alerts the test framework that the tests should be run. + This ensures that the active tests and the nodes are run synchronously. +* An undecorated class inheriting from ``unittest.TestCase``: + houses the active tests, including set up and teardown. + One has access to the ROS logging through ``proc_output``. +* A second class inheriting from ``unittest.TestCase``, + decorated with ``@launch_testing.post_shutdown_test()``. + As the name implies, these tests run after all nodes have shutdown. + A common assert here is to check the exit codes, + to ensure all nodes exited cleanly. + +The launch test is subsequently registered in the ``CMakeLists.txt`` +using the custom cmake macro ``add_ros_isolated_launch_test`` +that ensures that each launch test runs with a unique ``ROS_DOMAIN_ID``, +avoiding undesired cross communication. + +To finish, tools such as Xunit Viewer ease visualizing the test results +in a more colorful way than the colcon utilities. +We hope this tutorial gave the reader a good grasp +of how to conduct integration tests in ROS 2, +delineating tests, and analyzing their results. + + +Next steps +---------- +* Extend the two basic active tests with one that checks + whether the turtle is responsive to twist commands, + and another that verifies whether the spawn service + sets the turtle to the intended pose + (see the `Turtlesim introduction tutorial <../../Beginner-CLI-Tools/Introducing-Turtlesim/Introducing-Turtlesim>`). +* Instead of Turtlesim, launch the + :doc:`Gazebo simulator <../../Advanced/Simulators/Gazebo/Gazebo>` + and simulate *your* robot in there, automating tests + that would otherwise depend on manually operating your physical robot. +* Go through the + `launch_testing documentation `_ + and explore the many possibilities for integration testing it offers. + + +Related content +--------------- + +* :doc:`Why automatic tests? <../../Intermediate/Testing/Testing-Main>` +* :doc:`C++ unit testing with GTest <../../Intermediate/Testing/Cpp>` + and :doc:`Python unit testing with Pytest <../../Intermediate/Testing/Python>` +* `launch_pytest documentation `_, + an alternative launch integration testing package to ``launch_testing`` diff --git a/source/Tutorials/Intermediate/Testing/Testing-Main.rst b/source/Tutorials/Intermediate/Testing/Testing-Main.rst index 04985c7b99d..2f905a7f749 100644 --- a/source/Tutorials/Intermediate/Testing/Testing-Main.rst +++ b/source/Tutorials/Intermediate/Testing/Testing-Main.rst @@ -41,4 +41,5 @@ Available Tutorials: CLI Cpp Python + Integration BuildFarmTesting From b3e11d176fc9dd3b0b8b55584ad9521ec942e984 Mon Sep 17 00:00:00 2001 From: abaeyens Date: Sat, 7 Dec 2024 15:22:21 +0100 Subject: [PATCH 02/15] tutorial/integration_testing: apply suggestions from code review Thanks for the many suggestions and feedback! Applying the suggestions in multiple batches... Co-authored-by: Chris Lalancette Co-authored-by: Tomoya Fujita Signed-off-by: abaeyens --- .../Intermediate/Testing/Integration.rst | 40 ++++++------------- 1 file changed, 12 insertions(+), 28 deletions(-) diff --git a/source/Tutorials/Intermediate/Testing/Integration.rst b/source/Tutorials/Intermediate/Testing/Integration.rst index eafd854dd30..24a0b6ad1f8 100644 --- a/source/Tutorials/Intermediate/Testing/Integration.rst +++ b/source/Tutorials/Intermediate/Testing/Integration.rst @@ -1,9 +1,7 @@ -.. TestingIntegration: - Writing Basic Integration Tests with launch_testing =================================================== -**Goal:** Create and run integration tests on the ROS 2 Turtlesim node. +**Goal:** Create and run integration tests on the ROS 2 turtlesim node. **Tutorial level:** Intermediate @@ -16,8 +14,7 @@ Writing Basic Integration Tests with launch_testing Prerequisites ------------- -Before starting this tutorial, -it is recommended to have completed the following tutorials on launching nodes: +Before starting this tutorial, it is recommended to have completed the following tutorials on launching nodes: * :doc:`Launching Multiple Nodes <../../Beginner-CLI-Tools/Launching-Multiple-Nodes/Launching-Multiple-Nodes>` * :doc:`Creating Launch files <../../Intermediate/Launch/Creating-Launch-Files>` @@ -25,16 +22,13 @@ it is recommended to have completed the following tutorials on launching nodes: Background ---------- -Where unit tests focus on validating a very specific piece of functionality, -integration tests go all the way in the other direction. -In ROS 2 this means launching a system of one or several nodes, -for example the Gazebo simulator and the Nav2 navigation stack. +Where unit tests focus on validating a very specific piece of functionality, integration tests focus on validating the interaction between pieces of code. +In ROS 2 this is often accomplished by launching a system of one or several nodes, for example the `Gazebo simulator `__ and the `Nav2 navigation `__ stack. As a result, these tests are more complex both to set up and to run. -A key aspect of ROS 2 integration testing is that nodes part of different tests -shouldn't communicate with each other, even when run in parallel, -which will be achieved here using a specific test runner -that picks unique ROS domain id's. +A key aspect of ROS 2 integration testing is that nodes that are part of different tests +shouldn't communicate with each other, even when run in parallel. +This will be achieved here using a specific test runner that picks unique :doc:`ROS domain IDs <../../../Concepts/Intermediate/About-Domain-ID>`. In addition, integration tests have to fit in the overall testing workflow. A standardized approach is to ensure each test outputs an XUnit file, which are easily parsed using common test tooling. @@ -42,27 +36,17 @@ which are easily parsed using common test tooling. Overview -------- -The main tool in use here is the -`launch_testing `_ -package +The main tool in use here is the `launch_testing `_ package (`launch_testing repository `_). -This ROS-agnostic functionality allows to extend a classic Python launch file +This ROS-agnostic functionality can extend a Python launch file with both active tests (that run while the nodes are also running) and post-shutdown tests (which run once after all nodes have exited). ``launch_testing`` relies on the Python standard module `unittest `_ for the actual testing. -Hence, preferably avoid mixing with ``pytest``. -If you prefer using pytest for integration tests, -you will want to use the package -`launch_pytest `_ -(`launch_pytest repository `_), -though this tutorial will discuss using unittest. - -To get our integration tests run as part of ``colcon test``, -we register the launch file in the ``CMakeLists.txt``. -To finish, we will briefly touch -on how to run these tests and inspect the test results. + +To get our integration tests run as part of ``colcon test``, we register the launch file in the ``CMakeLists.txt``. +To finish, we will briefly touch on how to run these tests and inspect the test results. Steps From 0a01d387f9860795b140f659e6acd5ad7f032f8c Mon Sep 17 00:00:00 2001 From: abaeyens Date: Sat, 7 Dec 2024 15:23:17 +0100 Subject: [PATCH 03/15] tutorial/integration_testing: apply suggestions from code review Co-authored-by: Chris Lalancette Co-authored-by: Katherine Scott Signed-off-by: abaeyens --- .../Intermediate/Testing/Integration.rst | 52 ++++++------------- 1 file changed, 17 insertions(+), 35 deletions(-) diff --git a/source/Tutorials/Intermediate/Testing/Integration.rst b/source/Tutorials/Intermediate/Testing/Integration.rst index 24a0b6ad1f8..1dbfa0ba885 100644 --- a/source/Tutorials/Intermediate/Testing/Integration.rst +++ b/source/Tutorials/Intermediate/Testing/Integration.rst @@ -54,22 +54,17 @@ Steps 1 Describe the test in the test launch file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Both the nodes under test and the tests themselves are launched -using a Python launch file, which resembles a classic ROS 2 Python launch file. -It is custom to let the integration test launch file names -follow the pattern ``test/test_*.py``. - -There are two common types of tests in integration testing: -active tests, which run while the nodes under test are running, -and post-shutdown tests, which are run after exiting the nodes. +Both the nodes under test and the tests themselves are launched using a Python launch file, which resembles a ROS 2 Python launch file. +It is customary to make the integration test launch file names follow the pattern ``test/test_*.py``. + +There are two common types of tests in integration testing: active tests, which run while the nodes under test are running, and post-shutdown tests, which are run after exiting the nodes. We will cover both in this tutorial. 1.1 Imports ~~~~~~~~~~~ -The top imports the Python modules we will be using. -Only two modules are specific to testing: -the general-purpose ``unittest``, and ``launch_testing``. +We first start by importing the Python modules we will be using. +Only two modules are specific to testing: the general-purpose ``unittest``, and ``launch_testing``. .. code-block:: python @@ -87,15 +82,13 @@ the general-purpose ``unittest``, and ``launch_testing``. 1.2 Generate the test description ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The function ``generate_test_description`` describes what to launch, -similar to ``generate_launch_description`` -in a classic ROS 2 Python launch file. -In the example below, we launch the turtlesim node -and half a second later our tests. +The function ``generate_test_description`` describes what to launch, similar to ``generate_launch_description`` +in a ROS 2 Python launch file. +In the example below, we launch the turtlesim node and half a second later our tests. In more complex integration test setups, you will probably want to launch a system of several nodes, together with additional nodes -that performing mocking or must otherwise interact with the nodes under test. +that perform mocking or must otherwise interact with the nodes under test. Including existing (XML) launch files - to avoid launch file duplication - is supported. @@ -122,30 +115,19 @@ is supported. 1.3 Active tests ~~~~~~~~~~~~~~~~ -The active tests interact with the running nodes. In this tutorial, -we will check whether the Turtlesim node publishes pose messages -(by listening to the node's 'turtle1/pose' topic) -and whether it logs that it spawned the turtle -(by listening to stderr). +The active tests interact with the running nodes. +In this tutorial, we will check whether the Turtlesim node publishes pose messages (by listening to the node's 'turtle1/pose' topic) +and whether it logs that it spawned the turtle (by listening to stderr). The active tests are defined as methods of a class inheriting from `unittest.TestCase `_. The child class, here ``TestTurtleSim``, contains the following methods: -- ``test_*``: - the test methods, each performing some ROS communication - with the nodes under test and/or listening to the process output - (passed in through ``proc_output``). +- ``test_*``: the test methods, each performing some ROS communication with the nodes under test and/or listening to the process output (passed in through ``proc_output``). They are executed sequentially. -- ``setUp``, ``tearDown``: - respectively run before (to prepare the test fixture) - and after executing each test method. - By creating the node in the ``setUp`` method, - we use a different node instance for each test - to reduce the risk of tests contaminating each other. -- ``setUpClass``, ``tearDownClass``: - these class methods respectively run once before and after - executing all the test methods. +- ``setUp``, ``tearDown``: respectively run before (to prepare the test fixture) and after executing each test method. + By creating the node in the ``setUp`` method, we use a different node instance for each test to reduce the risk of tests communicating with each other. +- ``setUpClass``, ``tearDownClass``: these class methods respectively run once before and after executing all the test methods. It's highly recommended to go through `launch_testing's detailed documentation on this topic `_. From 2ded82fdf620d20aae4e67744571e7c4648d006a Mon Sep 17 00:00:00 2001 From: abaeyens Date: Sat, 7 Dec 2024 15:24:09 +0100 Subject: [PATCH 04/15] tutorial/integration_testing: apply suggestions from code review Co-authored-by: Chris Lalancette Signed-off-by: abaeyens --- .../Intermediate/Testing/Integration.rst | 64 ++++++------------- 1 file changed, 20 insertions(+), 44 deletions(-) diff --git a/source/Tutorials/Intermediate/Testing/Integration.rst b/source/Tutorials/Intermediate/Testing/Integration.rst index 1dbfa0ba885..68d049f402f 100644 --- a/source/Tutorials/Intermediate/Testing/Integration.rst +++ b/source/Tutorials/Intermediate/Testing/Integration.rst @@ -173,29 +173,17 @@ It's highly recommended to go through 'Spawning turtle [turtle1] at x=', timeout=5, stream='stderr') -Note that the way we listen to the 'turtle1/pose' topic -in ``test_publishes_pose`` differs from -:doc:`the usual approach <../../Beginner-Client-Libraries/Writing-A-Simple-Py-Publisher-And-Subscriber>`. -Instead of calling the blocking ``rclpy.spin``, -we trigger the ``spin_once`` method - -which executes the first open callback -(so, our subscriber callback if a message arrived within 1 s) - -until we have gathered all messages published over the last 10 s. - -If you want to go further, you can implement yourself a third test -that publishes a twist message, asking the turtle to move, -and subsequently checks that it moved -by asserting that the pose message changed, -effectively automating part of the -`Turtlesim introduction tutorial <../../Beginner-CLI-Tools/Introducing-Turtlesim/Introducing-Turtlesim>`. +Note that the way we listen to the 'turtle1/pose' topic in ``test_publishes_pose`` differs from :doc:`the usual approach <../../Beginner-Client-Libraries/Writing-A-Simple-Py-Publisher-And-Subscriber>`. +Instead of calling the blocking ``rclpy.spin``, we trigger the ``spin_once`` method - which executes the first available callback (our subscriber callback if a message arrived within 1 second) - until we have gathered all messages published over the last 10 seconds. + +If you want to go further, you can implement a third test that publishes a twist message, asking the turtle to move, and subsequently checks that it moved by asserting that the pose message changed, +effectively automating part of the `Turtlesim introduction tutorial <../../Beginner-CLI-Tools/Introducing-Turtlesim/Introducing-Turtlesim>`. 1.4 Post-shutdown tests ~~~~~~~~~~~~~~~~~~~~~~~ -The classes marked with the ``launch_testing.post_shutdown_test`` decorator -are run after letting the nodes under test exit. -A typical test here is whether the nodes exited cleanly, -for which ``launch_testing`` provides the method +The classes marked with the ``launch_testing.post_shutdown_test`` decorator are run after letting the nodes under test exit. +A typical test here is whether the nodes exited cleanly, for which ``launch_testing`` provides the method `asserts.assertExitCodes `_. .. code-block:: python @@ -217,9 +205,7 @@ and it also allows to specify *how* the test is to be run - in this case, with a unique domain id to ensure test isolation. This latter aspect is realized using the special test runner `run_test_isolated.py `_. -To ease adding several integration tests, -we define the CMake function ``add_ros_isolated_launch_test`` -such that each additional test requires only a single line. +To ease adding several integration tests, we define the CMake function ``add_ros_isolated_launch_test`` such that each additional test requires only a single line. .. code-block:: cmake @@ -244,8 +230,7 @@ such that each additional test requires only a single line. 3 Dependencies and package organization ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Finally, to avoid surprises, -add the following dependencies to your ``package.xml``: +Finally, add the following dependencies to your ``package.xml``: .. code-block:: XML @@ -258,8 +243,7 @@ add the following dependencies to your ``package.xml``: turtlesim -After following the above steps, your package (here named 'app') -ought to look as follows: +After following the above steps, your package (here named 'app') ought to look as follows: .. code-block:: @@ -269,24 +253,18 @@ ought to look as follows: tests/ test_integration.py -Concerning package organization: Integration tests can be part of any ROS package. -One can dedicate one or more packages to just integration testing, -or alternatively add them to the package of which they test the functionality. -In this tutorial, we go with the first option -as we will test the existing Turtlesim node. +One can dedicate one or more packages to just integration testing, or alternatively add them to the package of which they test the functionality. +In this tutorial, we go with the first option as we will test the existing Turtlesim node. 4 Running tests and report generation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 4.1 Running with colcon ~~~~~~~~~~~~~~~~~~~~~~~ -Running all tests is straightforward: simply run -:doc:`colcon test <../../Intermediate/Testing/CLI>`. -This command suppresses the test output -and exposes little about which tests succeed and which fail. -Useful therefore while developing tests is the option -to print all test output while the tests are running: +Running all tests is straightforward: simply run :doc:`colcon test <../../Intermediate/Testing/CLI>`. +This command suppresses the test output and exposes little about which tests succeed and which fail. +Therefore while developing tests the ``--event-handlers`` option is useful to print all test output while the tests are running: .. code-block:: console @@ -305,13 +283,11 @@ For viewing the results, there's a separate colcon verb. For example, Summary: 4 tests, 0 errors, 2 failures, 0 skipped -lists two files: -one ctest-formatted XML file (a result of the ``CMakeLists.txt``) and, -more interestingly, also an XUnit-formatted XML file. -This latter is suitable for automatic report generation -in automated testing in CI/CD pipelines. -If we would have also added unit tests, their XUnit files -would show up as well here. +This command lists two files: +* a ctest-formatted XML file (a result of the ``CMakeLists.txt``) +* an XUnit-formatted XML file, which is suitable for automatic report generation in automated testing in CI/CD pipelines. + +If we would have also added unit tests, their XUnit files would show up as well here. A suitable tool to visualize them all together is the `NodeJS package Xunit Viewer `_. From d134ecea4fd12e00388a468888bea4b596055e6a Mon Sep 17 00:00:00 2001 From: abaeyens Date: Sat, 7 Dec 2024 15:25:03 +0100 Subject: [PATCH 05/15] tutorial/integration_testing: apply suggestions from code review Co-authored-by: Chris Lalancette Co-authored-by: Katherine Scott Signed-off-by: abaeyens --- .../Intermediate/Testing/Integration.rst | 40 +++++-------------- 1 file changed, 10 insertions(+), 30 deletions(-) diff --git a/source/Tutorials/Intermediate/Testing/Integration.rst b/source/Tutorials/Intermediate/Testing/Integration.rst index 68d049f402f..7c7b3e2c0e2 100644 --- a/source/Tutorials/Intermediate/Testing/Integration.rst +++ b/source/Tutorials/Intermediate/Testing/Integration.rst @@ -314,38 +314,18 @@ For example, command and response (without highlighting): Summary ------- -In this tutorial, we explored the process of creating and running -integration tests on the ROS 2 Turtlesim node. -We discussed the integration test launch file -and covered writing active tests and post-shutdown tests. +In this tutorial, we explored the process of creating and running integration tests on the ROS 2 Turtlesim node. +We discussed the integration test launch file and covered writing active tests and post-shutdown tests. To recap, the four key elements of the integration test launch file are: -* The function ``generate_test_description``: - same as the classic way of launching nodes - (basically, it replaces ``generate_launch_description``). - It launches our nodes under tests as well as our tests. -* ``launch_testing.actions.ReadyToTest()``: - alerts the test framework that the tests should be run. - This ensures that the active tests and the nodes are run synchronously. -* An undecorated class inheriting from ``unittest.TestCase``: - houses the active tests, including set up and teardown. - One has access to the ROS logging through ``proc_output``. -* A second class inheriting from ``unittest.TestCase``, - decorated with ``@launch_testing.post_shutdown_test()``. - As the name implies, these tests run after all nodes have shutdown. - A common assert here is to check the exit codes, - to ensure all nodes exited cleanly. - -The launch test is subsequently registered in the ``CMakeLists.txt`` -using the custom cmake macro ``add_ros_isolated_launch_test`` -that ensures that each launch test runs with a unique ``ROS_DOMAIN_ID``, +* The function ``generate_test_description``: This launches our nodes under tests as well as our tests. +* ``launch_testing.actions.ReadyToTest()``: This alerts the test framework that the tests should be run, and ensures that the active tests and the nodes are run together. +* An undecorated class inheriting from ``unittest.TestCase``: This houses the active tests, including set up and teardown, and gives access to ROS logging through ``proc_output``. +* A second class inheriting from ``unittest.TestCase`` decorated with ``@launch_testing.post_shutdown_test()``: These are tests that run after all nodes have shutdown; it is common to assert that the nodes exited cleanly. + +The launch test is subsequently registered in the ``CMakeLists.txt`` using the custom cmake macro ``add_ros_isolated_launch_test`` which ensures that each launch test runs with a unique ``ROS_DOMAIN_ID``, avoiding undesired cross communication. -To finish, tools such as Xunit Viewer ease visualizing the test results -in a more colorful way than the colcon utilities. -We hope this tutorial gave the reader a good grasp -of how to conduct integration tests in ROS 2, -delineating tests, and analyzing their results. Next steps @@ -354,8 +334,8 @@ Next steps whether the turtle is responsive to twist commands, and another that verifies whether the spawn service sets the turtle to the intended pose - (see the `Turtlesim introduction tutorial <../../Beginner-CLI-Tools/Introducing-Turtlesim/Introducing-Turtlesim>`). -* Instead of Turtlesim, launch the + (see the `turtlesim introduction tutorial <../../Beginner-CLI-Tools/Introducing-Turtlesim/Introducing-Turtlesim>`). +* Instead of turtlesim, launch the :doc:`Gazebo simulator <../../Advanced/Simulators/Gazebo/Gazebo>` and simulate *your* robot in there, automating tests that would otherwise depend on manually operating your physical robot. From b572725ba1eb63e66cf4b7c6c7094839e3f8f6b8 Mon Sep 17 00:00:00 2001 From: Arne Baeyens Date: Sat, 7 Dec 2024 15:27:36 +0100 Subject: [PATCH 06/15] tutorial/integration_testing: fix turtlesim capitalization Signed-off-by: Arne Baeyens --- source/Tutorials/Intermediate/Testing/Integration.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/source/Tutorials/Intermediate/Testing/Integration.rst b/source/Tutorials/Intermediate/Testing/Integration.rst index 7c7b3e2c0e2..891d2e48057 100644 --- a/source/Tutorials/Intermediate/Testing/Integration.rst +++ b/source/Tutorials/Intermediate/Testing/Integration.rst @@ -116,7 +116,7 @@ is supported. 1.3 Active tests ~~~~~~~~~~~~~~~~ The active tests interact with the running nodes. -In this tutorial, we will check whether the Turtlesim node publishes pose messages (by listening to the node's 'turtle1/pose' topic) +In this tutorial, we will check whether the turtlesim node publishes pose messages (by listening to the node's 'turtle1/pose' topic) and whether it logs that it spawned the turtle (by listening to stderr). The active tests are defined as methods of a class inheriting @@ -255,7 +255,7 @@ After following the above steps, your package (here named 'app') ought to look a Integration tests can be part of any ROS package. One can dedicate one or more packages to just integration testing, or alternatively add them to the package of which they test the functionality. -In this tutorial, we go with the first option as we will test the existing Turtlesim node. +In this tutorial, we go with the first option as we will test the existing turtlesim node. 4 Running tests and report generation @@ -314,7 +314,7 @@ For example, command and response (without highlighting): Summary ------- -In this tutorial, we explored the process of creating and running integration tests on the ROS 2 Turtlesim node. +In this tutorial, we explored the process of creating and running integration tests on the ROS 2 turtlesim node. We discussed the integration test launch file and covered writing active tests and post-shutdown tests. To recap, the four key elements of the integration test launch file are: From 07e941549049e2be0a1f207272f844ab2cad759e Mon Sep 17 00:00:00 2001 From: Arne Baeyens Date: Sat, 7 Dec 2024 15:40:25 +0100 Subject: [PATCH 07/15] tutorial/integration_testing: remove "homework" section and trailing whitespace Signed-off-by: Arne Baeyens --- .../Intermediate/Testing/Integration.rst | 21 ++----------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/source/Tutorials/Intermediate/Testing/Integration.rst b/source/Tutorials/Intermediate/Testing/Integration.rst index 891d2e48057..1cce6618fce 100644 --- a/source/Tutorials/Intermediate/Testing/Integration.rst +++ b/source/Tutorials/Intermediate/Testing/Integration.rst @@ -277,7 +277,7 @@ For viewing the results, there's a separate colcon verb. For example, .. code-block:: console - $ colcon test-result --all + $ colcon test-result --all build/app/Testing/20241013-0810/Test.xml: 1 tests, 0 errors, 1 failure, 0 skipped build/app/test_results/app/test_test_integration.py.xunit.xml: 3 tests, 0 errors, 1 failure, 0 skipped @@ -314,7 +314,7 @@ For example, command and response (without highlighting): Summary ------- -In this tutorial, we explored the process of creating and running integration tests on the ROS 2 turtlesim node. +In this tutorial, we explored the process of creating and running integration tests on the ROS 2 turtlesim node. We discussed the integration test launch file and covered writing active tests and post-shutdown tests. To recap, the four key elements of the integration test launch file are: @@ -327,23 +327,6 @@ The launch test is subsequently registered in the ``CMakeLists.txt`` using the c avoiding undesired cross communication. - -Next steps ----------- -* Extend the two basic active tests with one that checks - whether the turtle is responsive to twist commands, - and another that verifies whether the spawn service - sets the turtle to the intended pose - (see the `turtlesim introduction tutorial <../../Beginner-CLI-Tools/Introducing-Turtlesim/Introducing-Turtlesim>`). -* Instead of turtlesim, launch the - :doc:`Gazebo simulator <../../Advanced/Simulators/Gazebo/Gazebo>` - and simulate *your* robot in there, automating tests - that would otherwise depend on manually operating your physical robot. -* Go through the - `launch_testing documentation `_ - and explore the many possibilities for integration testing it offers. - - Related content --------------- From 5f46b89d24ef38fbe6792b78a7e07a45250be0aa Mon Sep 17 00:00:00 2001 From: Arne Baeyens Date: Sat, 7 Dec 2024 17:24:16 +0100 Subject: [PATCH 08/15] tutorial/integration_testing: delete include launch file ref + split up sentence Signed-off-by: Arne Baeyens --- .../Tutorials/Intermediate/Testing/Integration.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/source/Tutorials/Intermediate/Testing/Integration.rst b/source/Tutorials/Intermediate/Testing/Integration.rst index 1cce6618fce..0fc865bcb39 100644 --- a/source/Tutorials/Intermediate/Testing/Integration.rst +++ b/source/Tutorials/Intermediate/Testing/Integration.rst @@ -89,8 +89,6 @@ In the example below, we launch the turtlesim node and half a second later our t In more complex integration test setups, you will probably want to launch a system of several nodes, together with additional nodes that perform mocking or must otherwise interact with the nodes under test. -Including existing (XML) launch files - to avoid launch file duplication - -is supported. .. code-block:: python @@ -199,10 +197,12 @@ A typical test here is whether the nodes exited cleanly, for which ``launch_test 2 Register the test in the CMakeLists.txt ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Registering the test in the ``CMakeLists.txt`` fulfills two functions: -it integrates it in the ``CTest`` framework ROS 2 CMake-based packages rely on -(and hence it will be called when running ``colcon test``), -and it also allows to specify *how* the test is to be run - -in this case, with a unique domain id to ensure test isolation. + +- it integrates it in the ``CTest`` framework ROS 2 CMake-based packages rely on + (and hence it will be called when running ``colcon test``). +- it allows to specify *how* the test is to be run - + in this case, with a unique domain id to ensure test isolation. + This latter aspect is realized using the special test runner `run_test_isolated.py `_. To ease adding several integration tests, we define the CMake function ``add_ros_isolated_launch_test`` such that each additional test requires only a single line. From 05f16a7cc7d5f280dd6bc119a5d5410fe0179138 Mon Sep 17 00:00:00 2001 From: Arne Baeyens Date: Sat, 7 Dec 2024 17:39:57 +0100 Subject: [PATCH 09/15] tutorial/integration_testing: move test running and examination to existing tutorial Suggestion, or prefer to put this somewhere else? Signed-off-by: Arne Baeyens --- source/Tutorials/Intermediate/Testing/CLI.rst | 41 ++++++++++++++- .../Intermediate/Testing/Integration.rst | 52 +------------------ 2 files changed, 40 insertions(+), 53 deletions(-) diff --git a/source/Tutorials/Intermediate/Testing/CLI.rst b/source/Tutorials/Intermediate/Testing/CLI.rst index d8b83f2053f..292b2d17d40 100644 --- a/source/Tutorials/Intermediate/Testing/CLI.rst +++ b/source/Tutorials/Intermediate/Testing/CLI.rst @@ -17,14 +17,30 @@ To compile and run the tests, simply run the `test ` before testing should not be necessary. ``colcon test`` makes sure that the tests run with the right environment, have access to their dependencies, etc. +The above command suppresses the test output and exposes little about which tests succeed and which fail. +Therefore while developing tests the ``--event-handlers`` option is useful to print all test output while the tests are running: + +.. code-block:: console + + colcon test --event-handlers console_direct+ + Examine Test Results ^^^^^^^^^^^^^^^^^^^^ -To see the results, simply run the `test-result `__ verb from ``colcon``. +To see the results, simply run the `test-result `__ verb from ``colcon``: .. code-block:: console - colcon test-result --all + $ colcon test-result --all + build/app/Testing/20241013-0810/Test.xml: 1 tests, 0 errors, 1 failure, 0 skipped + build/app/test_results/app/test_test_myapp.py.xunit.xml: 3 tests, 0 errors, 1 failure, 0 skipped + + Summary: 4 tests, 0 errors, 2 failures, 0 skipped + +In the example above, the command lists two files: + +* a ctest-formatted XML file (a result of the ``CMakeLists.txt``) +* an XUnit-formatted XML file, which is suitable for automatic report generation in automated testing in CI/CD pipelines. To see the exact test cases which fail, use the ``--verbose`` flag: @@ -32,6 +48,27 @@ To see the exact test cases which fail, use the ``--verbose`` flag: colcon test-result --all --verbose +A suitable non-ROS specific tool to visualize them all together is the +`NodeJS package Xunit Viewer `_. +It converts the XUnit files to HTML or straight into the terminal. +For example, command and response (without highlighting): + +.. code-block:: console + + $ xunit-viewer -r build/app/test_results -c + app.test_integration.launch_tests + ✗ test_publishes_pose time=0.52 + - Traceback (most recent call last): + File "/home/user/ros_workspace/src/app/test/test_myapp.py", line 67, in test_publishes_pose + assert len(msgs_rx) > 100 + ^^^^^^^^^^^^^^^^^^ + AssertionError + ✓ test_exit_codes time=0.0 + ✓ test_logs_spawning time=0.197 + + 1 failure, 2 passed + Written to: /home/user/ros_workspace/index.html + Debugging tests with GDB ^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/source/Tutorials/Intermediate/Testing/Integration.rst b/source/Tutorials/Intermediate/Testing/Integration.rst index 0fc865bcb39..45cfda1b879 100644 --- a/source/Tutorials/Intermediate/Testing/Integration.rst +++ b/source/Tutorials/Intermediate/Testing/Integration.rst @@ -44,9 +44,7 @@ and post-shutdown tests (which run once after all nodes have exited). ``launch_testing`` relies on the Python standard module `unittest `_ for the actual testing. - To get our integration tests run as part of ``colcon test``, we register the launch file in the ``CMakeLists.txt``. -To finish, we will briefly touch on how to run these tests and inspect the test results. Steps @@ -260,55 +258,7 @@ In this tutorial, we go with the first option as we will test the existing turtl 4 Running tests and report generation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -4.1 Running with colcon -~~~~~~~~~~~~~~~~~~~~~~~ -Running all tests is straightforward: simply run :doc:`colcon test <../../Intermediate/Testing/CLI>`. -This command suppresses the test output and exposes little about which tests succeed and which fail. -Therefore while developing tests the ``--event-handlers`` option is useful to print all test output while the tests are running: - -.. code-block:: console - - colcon test --event-handlers console_direct+ - - -4.2 Visualizing test results -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -For viewing the results, there's a separate colcon verb. For example, - -.. code-block:: console - - $ colcon test-result --all - build/app/Testing/20241013-0810/Test.xml: 1 tests, 0 errors, 1 failure, 0 skipped - build/app/test_results/app/test_test_integration.py.xunit.xml: 3 tests, 0 errors, 1 failure, 0 skipped - - Summary: 4 tests, 0 errors, 2 failures, 0 skipped - -This command lists two files: -* a ctest-formatted XML file (a result of the ``CMakeLists.txt``) -* an XUnit-formatted XML file, which is suitable for automatic report generation in automated testing in CI/CD pipelines. - -If we would have also added unit tests, their XUnit files would show up as well here. - -A suitable tool to visualize them all together is the -`NodeJS package Xunit Viewer `_. -It converts the XUnit files to HTML or straight into the terminal. -For example, command and response (without highlighting): - -.. code-block:: console - - $ xunit-viewer -r build/app/test_results -c - app.test_integration.launch_tests - ✗ test_publishes_pose time=0.52 - - Traceback (most recent call last): - File "/home/user/ros_workspace/src/app/test/test_integration.py", line 67, in test_publishes_pose - assert len(msgs_rx) > 100 - ^^^^^^^^^^^^^^^^^^ - AssertionError - ✓ test_exit_codes time=0.0 - ✓ test_logs_spawning time=0.197 - - 1 failure, 2 passed - Written to: /home/user/ros_workspace/index.html +For running the integration test and examining the results, see the tutorial :doc:`Running Tests in ROS 2 from the Command Line<../../Intermediate/Testing/CLI>`. Summary From 13140f50b160c72c704823fc8d2ecbcd3473cfcc Mon Sep 17 00:00:00 2001 From: Arne Baeyens Date: Sat, 7 Dec 2024 17:45:58 +0100 Subject: [PATCH 10/15] tutorial/integration_testing: fix whitespace Follow https://docs.ros.org/en/rolling/The-ROS2-Project/Contributing/Code-Style-Language-Versions.html Signed-off-by: Arne Baeyens --- .../Intermediate/Testing/Integration.rst | 25 ++++++++----------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/source/Tutorials/Intermediate/Testing/Integration.rst b/source/Tutorials/Intermediate/Testing/Integration.rst index 45cfda1b879..b7904d7d2fb 100644 --- a/source/Tutorials/Intermediate/Testing/Integration.rst +++ b/source/Tutorials/Intermediate/Testing/Integration.rst @@ -11,17 +11,17 @@ Writing Basic Integration Tests with launch_testing :depth: 2 :local: - Prerequisites ------------- + Before starting this tutorial, it is recommended to have completed the following tutorials on launching nodes: * :doc:`Launching Multiple Nodes <../../Beginner-CLI-Tools/Launching-Multiple-Nodes/Launching-Multiple-Nodes>` * :doc:`Creating Launch files <../../Intermediate/Launch/Creating-Launch-Files>` - Background ---------- + Where unit tests focus on validating a very specific piece of functionality, integration tests focus on validating the interaction between pieces of code. In ROS 2 this is often accomplished by launching a system of one or several nodes, for example the `Gazebo simulator `__ and the `Nav2 navigation `__ stack. As a result, these tests are more complex both to set up and to run. @@ -33,9 +33,9 @@ In addition, integration tests have to fit in the overall testing workflow. A standardized approach is to ensure each test outputs an XUnit file, which are easily parsed using common test tooling. - Overview -------- + The main tool in use here is the `launch_testing `_ package (`launch_testing repository `_). This ROS-agnostic functionality can extend a Python launch file @@ -46,21 +46,21 @@ and post-shutdown tests (which run once after all nodes have exited). for the actual testing. To get our integration tests run as part of ``colcon test``, we register the launch file in the ``CMakeLists.txt``. - Steps ----- 1 Describe the test in the test launch file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + Both the nodes under test and the tests themselves are launched using a Python launch file, which resembles a ROS 2 Python launch file. It is customary to make the integration test launch file names follow the pattern ``test/test_*.py``. There are two common types of tests in integration testing: active tests, which run while the nodes under test are running, and post-shutdown tests, which are run after exiting the nodes. We will cover both in this tutorial. - 1.1 Imports ~~~~~~~~~~~ + We first start by importing the Python modules we will be using. Only two modules are specific to testing: the general-purpose ``unittest``, and ``launch_testing``. @@ -77,9 +77,9 @@ Only two modules are specific to testing: the general-purpose ``unittest``, and import rclpy from turtlesim.msg import Pose - 1.2 Generate the test description ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + The function ``generate_test_description`` describes what to launch, similar to ``generate_launch_description`` in a ROS 2 Python launch file. In the example below, we launch the turtlesim node and half a second later our tests. @@ -108,9 +108,9 @@ that perform mocking or must otherwise interact with the nodes under test. ), {}, ) - 1.3 Active tests ~~~~~~~~~~~~~~~~ + The active tests interact with the running nodes. In this tutorial, we will check whether the turtlesim node publishes pose messages (by listening to the node's 'turtle1/pose' topic) and whether it logs that it spawned the turtle (by listening to stderr). @@ -175,9 +175,9 @@ Instead of calling the blocking ``rclpy.spin``, we trigger the ``spin_once`` met If you want to go further, you can implement a third test that publishes a twist message, asking the turtle to move, and subsequently checks that it moved by asserting that the pose message changed, effectively automating part of the `Turtlesim introduction tutorial <../../Beginner-CLI-Tools/Introducing-Turtlesim/Introducing-Turtlesim>`. - 1.4 Post-shutdown tests ~~~~~~~~~~~~~~~~~~~~~~~ + The classes marked with the ``launch_testing.post_shutdown_test`` decorator are run after letting the nodes under test exit. A typical test here is whether the nodes exited cleanly, for which ``launch_testing`` provides the method `asserts.assertExitCodes `_. @@ -191,9 +191,9 @@ A typical test here is whether the nodes exited cleanly, for which ``launch_test """Check if the processes exited normally.""" launch_testing.asserts.assertExitCodes(proc_info) - 2 Register the test in the CMakeLists.txt ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + Registering the test in the ``CMakeLists.txt`` fulfills two functions: - it integrates it in the ``CTest`` framework ROS 2 CMake-based packages rely on @@ -225,9 +225,9 @@ To ease adding several integration tests, we define the CMake function ``add_ros add_ros_isolated_launch_test(test/test_integration.py) endif() - 3 Dependencies and package organization ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + Finally, add the following dependencies to your ``package.xml``: .. code-block:: XML @@ -240,7 +240,6 @@ Finally, add the following dependencies to your ``package.xml``: rclpy turtlesim - After following the above steps, your package (here named 'app') ought to look as follows: .. code-block:: @@ -255,11 +254,10 @@ Integration tests can be part of any ROS package. One can dedicate one or more packages to just integration testing, or alternatively add them to the package of which they test the functionality. In this tutorial, we go with the first option as we will test the existing turtlesim node. - 4 Running tests and report generation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -For running the integration test and examining the results, see the tutorial :doc:`Running Tests in ROS 2 from the Command Line<../../Intermediate/Testing/CLI>`. +For running the integration test and examining the results, see the tutorial :doc:`Running Tests in ROS 2 from the Command Line<../../Intermediate/Testing/CLI>`. Summary ------- @@ -276,7 +274,6 @@ To recap, the four key elements of the integration test launch file are: The launch test is subsequently registered in the ``CMakeLists.txt`` using the custom cmake macro ``add_ros_isolated_launch_test`` which ensures that each launch test runs with a unique ``ROS_DOMAIN_ID``, avoiding undesired cross communication. - Related content --------------- From c28db38b9ed00f5e54ef2e6504e793687829a719 Mon Sep 17 00:00:00 2001 From: Arne Baeyens Date: Sun, 8 Dec 2024 14:58:17 +0100 Subject: [PATCH 11/15] tutorial/integration_testing: link to launch_testing_ros package Signed-off-by: Arne Baeyens --- source/Tutorials/Intermediate/Testing/Integration.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/source/Tutorials/Intermediate/Testing/Integration.rst b/source/Tutorials/Intermediate/Testing/Integration.rst index b7904d7d2fb..4f0011b930f 100644 --- a/source/Tutorials/Intermediate/Testing/Integration.rst +++ b/source/Tutorials/Intermediate/Testing/Integration.rst @@ -171,6 +171,8 @@ It's highly recommended to go through Note that the way we listen to the 'turtle1/pose' topic in ``test_publishes_pose`` differs from :doc:`the usual approach <../../Beginner-Client-Libraries/Writing-A-Simple-Py-Publisher-And-Subscriber>`. Instead of calling the blocking ``rclpy.spin``, we trigger the ``spin_once`` method - which executes the first available callback (our subscriber callback if a message arrived within 1 second) - until we have gathered all messages published over the last 10 seconds. +The package `launch_testing_ros `_ provides some convenience functions to achieve similar behavior, +such as `WaitForTopics `_. If you want to go further, you can implement a third test that publishes a twist message, asking the turtle to move, and subsequently checks that it moved by asserting that the pose message changed, effectively automating part of the `Turtlesim introduction tutorial <../../Beginner-CLI-Tools/Introducing-Turtlesim/Introducing-Turtlesim>`. From b9278e7b56f11aeb43239d2602604a1fb4614549 Mon Sep 17 00:00:00 2001 From: Arne Baeyens Date: Sun, 8 Dec 2024 22:09:18 +0100 Subject: [PATCH 12/15] tutorial/integration_testing: fix path Should cover all packages, not just the 'app' package which may not even exist in the user's workspace. Signed-off-by: Arne Baeyens --- source/Tutorials/Intermediate/Testing/CLI.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/Tutorials/Intermediate/Testing/CLI.rst b/source/Tutorials/Intermediate/Testing/CLI.rst index 292b2d17d40..d8e352fb3ad 100644 --- a/source/Tutorials/Intermediate/Testing/CLI.rst +++ b/source/Tutorials/Intermediate/Testing/CLI.rst @@ -55,7 +55,7 @@ For example, command and response (without highlighting): .. code-block:: console - $ xunit-viewer -r build/app/test_results -c + $ xunit-viewer -r build -c app.test_integration.launch_tests ✗ test_publishes_pose time=0.52 - Traceback (most recent call last): From 620dbab56d8c479b98b220f1bb5ba143e3aaf1f2 Mon Sep 17 00:00:00 2001 From: Arne Baeyens Date: Mon, 9 Dec 2024 20:15:12 +0100 Subject: [PATCH 13/15] tutorial/integration_testing: Leave changes to CLI tutorial for other PR. Signed-off-by: Arne Baeyens --- source/Tutorials/Intermediate/Testing/CLI.rst | 41 +------------------ 1 file changed, 2 insertions(+), 39 deletions(-) diff --git a/source/Tutorials/Intermediate/Testing/CLI.rst b/source/Tutorials/Intermediate/Testing/CLI.rst index d8e352fb3ad..d8b83f2053f 100644 --- a/source/Tutorials/Intermediate/Testing/CLI.rst +++ b/source/Tutorials/Intermediate/Testing/CLI.rst @@ -17,30 +17,14 @@ To compile and run the tests, simply run the `test ` before testing should not be necessary. ``colcon test`` makes sure that the tests run with the right environment, have access to their dependencies, etc. -The above command suppresses the test output and exposes little about which tests succeed and which fail. -Therefore while developing tests the ``--event-handlers`` option is useful to print all test output while the tests are running: - -.. code-block:: console - - colcon test --event-handlers console_direct+ - Examine Test Results ^^^^^^^^^^^^^^^^^^^^ -To see the results, simply run the `test-result `__ verb from ``colcon``: +To see the results, simply run the `test-result `__ verb from ``colcon``. .. code-block:: console - $ colcon test-result --all - build/app/Testing/20241013-0810/Test.xml: 1 tests, 0 errors, 1 failure, 0 skipped - build/app/test_results/app/test_test_myapp.py.xunit.xml: 3 tests, 0 errors, 1 failure, 0 skipped - - Summary: 4 tests, 0 errors, 2 failures, 0 skipped - -In the example above, the command lists two files: - -* a ctest-formatted XML file (a result of the ``CMakeLists.txt``) -* an XUnit-formatted XML file, which is suitable for automatic report generation in automated testing in CI/CD pipelines. + colcon test-result --all To see the exact test cases which fail, use the ``--verbose`` flag: @@ -48,27 +32,6 @@ To see the exact test cases which fail, use the ``--verbose`` flag: colcon test-result --all --verbose -A suitable non-ROS specific tool to visualize them all together is the -`NodeJS package Xunit Viewer `_. -It converts the XUnit files to HTML or straight into the terminal. -For example, command and response (without highlighting): - -.. code-block:: console - - $ xunit-viewer -r build -c - app.test_integration.launch_tests - ✗ test_publishes_pose time=0.52 - - Traceback (most recent call last): - File "/home/user/ros_workspace/src/app/test/test_myapp.py", line 67, in test_publishes_pose - assert len(msgs_rx) > 100 - ^^^^^^^^^^^^^^^^^^ - AssertionError - ✓ test_exit_codes time=0.0 - ✓ test_logs_spawning time=0.197 - - 1 failure, 2 passed - Written to: /home/user/ros_workspace/index.html - Debugging tests with GDB ^^^^^^^^^^^^^^^^^^^^^^^^ From 2270b2e307a90bd1b35e3ccf50ef3553dfb97f41 Mon Sep 17 00:00:00 2001 From: abaeyens Date: Mon, 9 Dec 2024 20:16:06 +0100 Subject: [PATCH 14/15] Update source/Tutorials/Intermediate/Testing/Integration.rst Co-authored-by: Chris Lalancette Signed-off-by: abaeyens --- source/Tutorials/Intermediate/Testing/Integration.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/source/Tutorials/Intermediate/Testing/Integration.rst b/source/Tutorials/Intermediate/Testing/Integration.rst index 4f0011b930f..4f029ab62a5 100644 --- a/source/Tutorials/Intermediate/Testing/Integration.rst +++ b/source/Tutorials/Intermediate/Testing/Integration.rst @@ -174,8 +174,8 @@ Instead of calling the blocking ``rclpy.spin``, we trigger the ``spin_once`` met The package `launch_testing_ros `_ provides some convenience functions to achieve similar behavior, such as `WaitForTopics `_. -If you want to go further, you can implement a third test that publishes a twist message, asking the turtle to move, and subsequently checks that it moved by asserting that the pose message changed, -effectively automating part of the `Turtlesim introduction tutorial <../../Beginner-CLI-Tools/Introducing-Turtlesim/Introducing-Turtlesim>`. +If you want to go further, you can implement a third test that publishes a twist message, asking the turtle to move, and subsequently checks that it moved by asserting that the pose message changed. +This effectively automates part of the `Turtlesim introduction tutorial <../../Beginner-CLI-Tools/Introducing-Turtlesim/Introducing-Turtlesim>`. 1.4 Post-shutdown tests ~~~~~~~~~~~~~~~~~~~~~~~ From 29bc96695c29ccf54debc8d62c17c0ac97201a58 Mon Sep 17 00:00:00 2001 From: Chris Lalancette Date: Mon, 9 Dec 2024 14:31:05 -0500 Subject: [PATCH 15/15] Minor formatting fixes. Signed-off-by: Chris Lalancette --- .../Intermediate/Testing/Integration.rst | 33 ++++++------------- 1 file changed, 10 insertions(+), 23 deletions(-) diff --git a/source/Tutorials/Intermediate/Testing/Integration.rst b/source/Tutorials/Intermediate/Testing/Integration.rst index 4f029ab62a5..ac51c2d7c78 100644 --- a/source/Tutorials/Intermediate/Testing/Integration.rst +++ b/source/Tutorials/Intermediate/Testing/Integration.rst @@ -26,24 +26,18 @@ Where unit tests focus on validating a very specific piece of functionality, int In ROS 2 this is often accomplished by launching a system of one or several nodes, for example the `Gazebo simulator `__ and the `Nav2 navigation `__ stack. As a result, these tests are more complex both to set up and to run. -A key aspect of ROS 2 integration testing is that nodes that are part of different tests -shouldn't communicate with each other, even when run in parallel. +A key aspect of ROS 2 integration testing is that nodes that are part of different tests shouldn't communicate with each other, even when run in parallel. This will be achieved here using a specific test runner that picks unique :doc:`ROS domain IDs <../../../Concepts/Intermediate/About-Domain-ID>`. In addition, integration tests have to fit in the overall testing workflow. -A standardized approach is to ensure each test outputs an XUnit file, -which are easily parsed using common test tooling. +A standardized approach is to ensure each test outputs an XUnit file, which are easily parsed using common test tooling. Overview -------- The main tool in use here is the `launch_testing `_ package (`launch_testing repository `_). -This ROS-agnostic functionality can extend a Python launch file -with both active tests (that run while the nodes are also running) -and post-shutdown tests (which run once after all nodes have exited). -``launch_testing`` relies on the Python standard module -`unittest `_ -for the actual testing. +This ROS-agnostic functionality can extend a Python launch file with both active tests (that run while the nodes are also running) and post-shutdown tests (which run once after all nodes have exited). +``launch_testing`` relies on the Python standard module `unittest `_ for the actual testing. To get our integration tests run as part of ``colcon test``, we register the launch file in the ``CMakeLists.txt``. Steps @@ -80,13 +74,10 @@ Only two modules are specific to testing: the general-purpose ``unittest``, and 1.2 Generate the test description ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The function ``generate_test_description`` describes what to launch, similar to ``generate_launch_description`` -in a ROS 2 Python launch file. +The function ``generate_test_description`` describes what to launch, similar to ``generate_launch_description`` in a ROS 2 Python launch file. In the example below, we launch the turtlesim node and half a second later our tests. -In more complex integration test setups, you will probably want -to launch a system of several nodes, together with additional nodes -that perform mocking or must otherwise interact with the nodes under test. +In more complex integration test setups, you will probably want to launch a system of several nodes, together with additional nodes that perform mocking or must otherwise interact with the nodes under test. .. code-block:: python @@ -112,11 +103,9 @@ that perform mocking or must otherwise interact with the nodes under test. ~~~~~~~~~~~~~~~~ The active tests interact with the running nodes. -In this tutorial, we will check whether the turtlesim node publishes pose messages (by listening to the node's 'turtle1/pose' topic) -and whether it logs that it spawned the turtle (by listening to stderr). +In this tutorial, we will check whether the turtlesim node publishes pose messages (by listening to the node's 'turtle1/pose' topic) and whether it logs that it spawned the turtle (by listening to stderr). -The active tests are defined as methods of a class inheriting -from `unittest.TestCase `_. +The active tests are defined as methods of a class inheriting from `unittest.TestCase `_. The child class, here ``TestTurtleSim``, contains the following methods: - ``test_*``: the test methods, each performing some ROS communication with the nodes under test and/or listening to the process output (passed in through ``proc_output``). @@ -125,8 +114,7 @@ The child class, here ``TestTurtleSim``, contains the following methods: By creating the node in the ``setUp`` method, we use a different node instance for each test to reduce the risk of tests communicating with each other. - ``setUpClass``, ``tearDownClass``: these class methods respectively run once before and after executing all the test methods. -It's highly recommended to go through -`launch_testing's detailed documentation on this topic `_. +It's highly recommended to go through `launch_testing's detailed documentation on this topic `_. .. code-block:: python @@ -203,8 +191,7 @@ Registering the test in the ``CMakeLists.txt`` fulfills two functions: - it allows to specify *how* the test is to be run - in this case, with a unique domain id to ensure test isolation. -This latter aspect is realized using the special test runner -`run_test_isolated.py `_. +This latter aspect is realized using the special test runner `run_test_isolated.py `_. To ease adding several integration tests, we define the CMake function ``add_ros_isolated_launch_test`` such that each additional test requires only a single line. .. code-block:: cmake