Junit5 test discovery reports spurious test runs for non-test classes (500USD bounty)

[lihaoyi]

---

From the maintainer Li Haoyi: I'm putting a 500USD bounty on this issue, payable by bank transfer on a merged PR implementing this.

---

First noticed when reviewing #3886

lihaoyi mill$ ./mill dist.run example/kotlinlib/basic/1-simple -i foo.test
[2617/2617] dist.run
======================================================== foo.test ====================================================
======================================================================================================================
[99/99] foo.test.test
[99] Warning: Kotest autoscan is enabled. This means Kotest will scan the classpath for extensions that are annotated with @AutoScan. To avoid this startup cost, disable autoscan by setting the system property 'kotest.framework.classpath.scanning.autoscan.disable=true'. In 6.0 this value will default to true. For further details see https://kotest.io/docs/next/framework/project-config.html#runtime-detection
[99] Test run started (Kotest)
[99] Test foo.FooTesttestSimple started
[99] Test foo.FooTesttestSimple finished, took 0.038s
[99] Test foo.FooTesttestEscaping started
[99] Test foo.FooTesttestEscaping finished, took 0.001s
[99] Test foo.FooTest finished, took 0.09s
[99] Test  finished, took 0.11s
[99] Test  finished, took 0.002s
[99] Test run finished: 0 failed, 0 ignored, 2 total, 0.14s
[99] Warning: Kotest autoscan is enabled. This means Kotest will scan the classpath for extensions that are annotated with @AutoScan. To avoid this startup cost, disable autoscan by setting the system property 'kotest.framework.classpath.scanning.autoscan.disable=true'. In 6.0 this value will default to true. For further details see https://kotest.io/docs/next/framework/project-config.html#runtime-detection
[99] Test  finished, took 0.001s
[99] Test  finished, took 0.0s
[99] Test run finished: 0 failed, 0 ignored, 0 total, 0.001s
[99] Warning: Kotest autoscan is enabled. This means Kotest will scan the classpath for extensions that are annotated with @AutoScan. To avoid this startup cost, disable autoscan by setting the system property 'kotest.framework.classpath.scanning.autoscan.disable=true'. In 6.0 this value will default to true. For further details see https://kotest.io/docs/next/framework/project-config.html#runtime-detection
[99] Test  finished, took 0.0s
[99] Test  finished, took 0.0s
[99] Test run finished: 0 failed, 0 ignored, 0 total, 0.001s
[99] Warning: Kotest autoscan is enabled. This means Kotest will scan the classpath for extensions that are annotated with @AutoScan. To avoid this startup cost, disable autoscan by setting the system property 'kotest.framework.classpath.scanning.autoscan.disable=true'. In 6.0 this value will default to true. For further details see https://kotest.io/docs/next/framework/project-config.html#runtime-detection
[99] Test  finished, took 0.0s
[99] Test  finished, took 0.0s
[99] Test run finished: 0 failed, 0 ignored, 0 total, 0.001s
[99/99] ================================================ foo.test ================================================= 7s
======================================================================================================================
[2617/2617] ======================= dist.run example/kotlinlib/basic/1-simple -i foo.test ============================ 8s

The line Test run finished: 0 failed, 0 ignored, 2 total is correct, but there also are three line Test run finished: 0 failed, 0 ignored, 0 total which are spurious. It seems to be something to do with the test framework incorrectly picking up the anonymous lambda classes as test classes, even though they have no tests

Also happens for non-kotlin tests, e.g. here is a standalone repro using Java/junit5 where the class Dummy causes a spurious Test run finished: 0 failed, 0 ignored, 0 total to be logged

lihaoyi mill$ git diff
diff --git a/example/javalib/testing/3-integration-suite/qux/test/src/qux/QuxTests.java b/example/javalib/testing/3-integration-suite/qux/test/src/qux/QuxTests.java
index 579a2eb55a..fa18297de3 100644
--- a/example/javalib/testing/3-integration-suite/qux/test/src/qux/QuxTests.java
+++ b/example/javalib/testing/3-integration-suite/qux/test/src/qux/QuxTests.java
@@ -16,4 +16,5 @@ public class QuxTests {
     String result = Qux.hello();
     assertTrue(result.endsWith("World"));
   }
-}
\ No newline at end of file
+}
+class Dummy{}
lihaoyi mill$ ./mill dist.run example/javalib/testing/3-integration-suite -i show qux.test
[2617/2617] dist.run
====================================================== show qux.test =================================================
======================================================================================================================
[1/1] show
[1-83/83] qux.test.test
[1-83] Test run started (JUnit Jupiter)
[1-83] Test qux.QuxTests#hello() started
[1-83] Test qux.QuxTests#hello() finished, took 0.012s
[1-83] Test qux.QuxTests#world() started
[1-83] Test qux.QuxTests#world() finished, took 0.001s
[1-83] Test qux.QuxTests finished, took 0.019s
[1-83] Test  finished, took 0.027s
[1-83] Test run finished: 0 failed, 0 ignored, 2 total, 0.037s
[1-83] Test  finished, took 0.001s
[1-83] Test run finished: 0 failed, 0 ignored, 0 total, 0.001s
[
  "",
  [
    {
      "fullyQualifiedName": "qux.QuxTests",
      "selector": "hello()",
      "duration": 11,
      "status": "Success"
    },
    {
      "fullyQualifiedName": "qux.QuxTests",
      "selector": "world()",
      "duration": 1,
      "status": "Success"
    }
  ]
]
[83/83] ============================================== show qux.test ============================================== 1s
[1] show
[1-83] qux.test.test
======================================================================================================================
[83/83] ============================================== show qux.test ============================================== 1s
======================================================================================================================
[2617/2617] =============== dist.run example/javalib/testing/3-integration-suite -i show qux.test ==================== 2s

[lefou]

With Mill snapshot release 0.12.1-43-9d1196 (which contains PR #3911) the situation is better, but I still see some empty test runs reported.

[lihaoyi]

Yes, because currently all classes in the test module are treated as test classes, regardless of the presence of superclass or annotation.

https://github.com/sbt/sbt-jupiter-interface says

sbt-jupiter-interface: sbt plugin. The sbt plugin makes the runtime library available to the sbt build by adding it to sbt.Keys.testFrameworks. It also overwrites sbt.Keys.detectTests with a custom task that uses JUnits discovery mechanism to collect available tests. This step is necessary since sbt is currently not capable of detecting package private test classes.

We probably need to see why they need to have special treatment for junit 5 in SBT, and whether we need to replicate that special treatment in Mill

[lihaoyi]

Here's the Jupiter custom test collector https://github.com/sbt/sbt-jupiter-interface/blob/468d4f31f1f6ce8529fff8a8804dd733974c7686/src/plugin/src/main/scala/com/github/sbt/junit/jupiter/sbt/JupiterPlugin.scala#L94C1-L119C1