Document that a TestExecutionListener cannot fail the TestPlan

[renyiwei-xinyi]

  <junit-jupiter.version>5.10.1</junit-jupiter.version>
  <junit-platform.version>1.10.1</junit-platform.version>

  public static void runSuit(TestContext context, Class<?> clazz){

        try (LauncherSession session = LauncherFactory.openSession()) {

            Launcher launcher = session.getLauncher();

            DiscoveryStart discoveryStart = new DiscoveryStart(clazz, context.getSteps().toArray(new String[0]));

            LauncherDiscoveryRequest request = discoveryStart.request();
            TestPlan testPlan = launcher.discover(request);

            ExecutionActionListenerPlus executionActionListenerPlus = new ExecutionActionListenerPlus();
            launcher.registerTestExecutionListeners(executionActionListenerPlus);

            LauncherDiscoveryListener launcherDiscoveryListener = LauncherDiscoveryListeners.abortOnFailure();
            launcher.registerLauncherDiscoveryListeners(launcherDiscoveryListener);

            SummaryGeneratingListener summaryGeneratingListener = new SummaryGeneratingListener();
            launcher.execute(request, summaryGeneratingListener);
        }
     }

  public class ExecutionActionListenerPlus implements TestExecutionListener {
    @Override
    public void executionFinished(TestIdentifier testIdentifier, TestExecutionResult testExecutionResult) {
        switch (testExecutionResult.getStatus()) {
            case SUCCESSFUL:
                break;
            case ABORTED:
                break;
            case FAILED:
                testExecutionResult.getThrowable().ifPresent((throwable) -> {
                    try {
                        throw throwable.getCause();
                    } catch (Throwable e) {
                        throw new TestException(e.getMessage());
                    }
                });
                break;
            default:
                break;
        }
    }
  }

I execute two test methods in a class, and expect the first test method to terminate execution if it encounters an exception, while the second test method will no longer execute; Actually, the second test method was still executed.

I think this breaks the agreement on the testing lifecycle.

My test component, some tests in a class have dependencies.I don't know how to solve this problem.

I think this is a bug because execution is uncontrollable and it wastes time and resources when executing large batches of cases

[sbrannen]

This is by design.

A TestExecutionListener cannot fail the TestPlan, and any exception thrown by a TestExecutionListener is caught and logged at WARN level.

See the following code for details.

junit5/junit-platform-launcher/src/main/java/org/junit/platform/launcher/core/CompositeTestExecutionListener.java

Lines 98 to 110 in 9115e23

	private static <T extends TestExecutionListener> void notifyEach(List<T> listeners, IterationOrder iterationOrder,
	Consumer<T> consumer, Supplier<String> description) {
	iterationOrder.forEach(listeners, listener -> {
	try {
	consumer.accept(listener);
	}
	catch (Throwable throwable) {
	UnrecoverableExceptions.rethrowIfUnrecoverable(throwable);
	logger.warn(throwable, () -> String.format("TestExecutionListener [%s] threw exception for method: %s",
	listener.getClass().getName(), description.get()));
	}
	});
	}

This behavior is similar to support for the TestWatcher API in JUnit Jupiter.

However, I can understand that this might not be intuitive.

I am therefore repurposing this ticket to improve the documentation regarding these semantics for the TestExecutionListener API.

[renyiwei-xinyi]

Oh~ Thanks. This is consistent with my conjecture after debugging until late at night 🤣
What should I do if I want to customize this design?
I just want to stop the subsequent execution that I think is useless when the case is abnormal 😭.

[sormuras]

Possibly related issue:

• Introduce pleaseStop() method for org.junit.platform.launcher.Launcher #1880