[junit-platform] Container failures don't cause Gradle failure

[kriegfrj]

The Gradle plugins (and I'm assuming the Maven ones too) rely on the tester returning a non-zero value from System.exit() to signify that at least one test failure has occurred.

The way it has been written, junit-platform tester returns the number of test failures, but if you have only container failures it will return zero. Such situations are not uncommon - here is an example (admittedly contrived) that illustrates the point:

class MyTestClass {
  @BeforeAll
  void beforeAll()  {
    throw new IllegalStateException();
  }

  @Test
  void myTest() {
  }
}

When this class is run, the beforeAll() will fail and Jupiter will register the test container has having failed. Moreover, it will skip all of the test methods, so myTest() will not register as a failure (or as a success for that matter). So the overall test result will be a single container failure and no test failures. Because junit-platform returns the test failure count and not the total failure count, the exit value will therefore be zero, meaning that Gradle will incorrectly conclude that the tests passed.

This is a significant issue, though probably not significant enough on its own to warrant a new release. If we are doing a new minor release at some point it would probably be worth cherry-picking this fix into it as the fix will be simple and low-risk.

[bjhargrave]

I just hit this in porting OSGi compliance tests to junit-platform. Some test cases relied upon Bnd calling the setBundleContext method (which junit-platform does not do) and then NPE'd in setUp because the context field had not been set. And no errors were reported!

[kriegfrj]

So sorry! Such an annoying bug, I'm a bit annoyed I didn't pick it up during development...

So do I interpret this as "let's do a 5.1.2 release"? 😄