Ability to run tests with a freshly launched system

[pbaughman]

Right now apex_launchtest works kind of like rostest from ros1. The system gets launched, all the tests run in alphabetical order, and then the system gets torn down.

We may want a way to specify that certain tests or maybe certain test cases (classes) should run against a freshly launched system. This will make the tests more unit-test like and prevent tests from interfering with one-another. The trade-off is performance. It will take extra time to start up and tear down the sytem

Some questions to answer about how this should work:

• How do post-shutdown tests fit into this? Do they need to run every time too?
• Can we use some sort of decorator that would go on individual tests or test classes that would tell the framework "hey, do another launch when you run these"

[hidmic]

We may want a way to specify that certain tests or maybe certain test cases (classes) should run against a freshly launched system.

Adding to this, I think that behavior should be the default.

There's a fundamental difference with rostest in that it didn't provide launch level testing tools but only test "nodes", effectively delegating testing to another process that may or may not be using another a testing framework like gtest, pytest, etc. (and there's already some work in that direction for ROS2 launch too, see ros2/launch#178)). For each test "node" found in a rostest file, the entire launch fixture is brought up and torn down.

This does not mean we can't have launch level testing reusing a given fixture, but that it should be explicitly opted-in by a user aware of the implications.

• How do post-shutdown tests fit into this? Do they need to run every time too?

If the testing framework runs a collection of collections of tests, each against it's own fixture, when post-shutdown tests are run boils down to in which collections a given test is found: in all, in a few, in its own.

• Can we use some sort of decorator that would go on individual tests or test classes that would tell the framework "hey, do another launch when you run these"

I don't know exactly how it'd look like, but I really like your decorator-based approach. It reminds me of pytest. So +1 to go down that route.

[pbaughman]

Spit-balling a little more on what this could look like in practice:

def generate_test_description(ready_fn):
  . . .

class Test1(unittest.TestCase):
    def test_alpha(self):
        . . .
    def test_bravo(self):
        . . .
    @post_shutdown_test
    def test_charlie(self):
        . . .

class Test2(unittest.TestCase):
    def test_delta(self):
        . . .
    @post_shutdown_test
    def test_echo(self):
        . . .

apex_launchtest would launch the stuff from the test description then run the alpha, bravo, and charlie tests from Test1 class. Then it would launch the stuff from the test description again and run the delta, and echo tests from the Test2 class

This doesn't make it the behavior by default, though.

[pbaughman]

I'm implementing PR #39 in a way that will make this issue a lot easier to complete. The 'magic' will only need to be in the loader.