WIP Platform Technology Compatibility Kit

[dotCipher]

Overview

Draft of refactoring the ExecutionRecorder to a separate package for public consumption. Includes event level timings on the resulting ExecutionGraph object returned from the call to execute(). Addresses issue #1356

---

I hereby agree to the terms of the JUnit Contributor License Agreement.

---

Definition of Done

• There are no TODOs left in the code
• Method preconditions are checked and documented in the method's Javadoc
• Coding conventions (e.g. for logging) have been followed
• Change is covered by automated tests
• Public API has Javadoc and @API annotations
• Change is documented in the User Guide and Release Notes
• All continuous integration builds pass

[dotCipher]

I realize this is a fairly large diff, so I figured it would be easier to chunk up the features that I was looking for. This PR aims to address the first requirement of #1356, but it sets up the framework for easier extenstion via the returned ExecutionGraph.

Ideally the last feature I would like to have is to return a List<TestExecution>, that can describe either the Test's successful / failure run, or a reason for it being skipped. I can add that in a follow up PR or this one if needed, but please advise :)

Lastly, I ran into an error locally with this branch that I wasn't sure how to fix:

> Task :platform-tests:test

org.junit.AutomaticModuleNameTests > automaticModuleName(String)[8] FAILED
    org.opentest4j.AssertionFailedError: test jar file failed: java.nio.file.NoSuchFileException: ../junit-platform-engine-test/build/libs/junit-platform-engine-test-1.2.0-SNAPSHOT.jar
        at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:40)
        at org.junit.jupiter.api.Assertions.fail(Assertions.java:82)
        at org.junit.AutomaticModuleNameTests.automaticModuleName(AutomaticModuleNameTests.java:105)

        Caused by:
        java.nio.file.NoSuchFileException: ../junit-platform-engine-test/build/libs/junit-platform-engine-test-1.2.0-SNAPSHOT.jar
            at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
            at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
            at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
            at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
            at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:145)
            at java.base/java.nio.file.Files.readAttributes(Files.java:1755)
            at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:951)
            at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:216)
            at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:148)
            at java.base/java.util.jar.JarFile.<init>(JarFile.java:324)
            at java.base/java.util.jar.JarFile.<init>(JarFile.java:295)
            at java.base/java.util.jar.JarFile.<init>(JarFile.java:261)
            at org.junit.AutomaticModuleNameTests.automaticModuleName(AutomaticModuleNameTests.java:84)
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/Users/cmoore/.gradle/caches/modules-2/files-2.1/org.codehaus.groovy/groovy/2.4.12/a43be367110c491787219f1c128b5b5fc54f1e70/groovy-2.4.12.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

There are WARNINGs from the build, as well as the failure marked above /\ Any insight on the steps I need to take would be appreciated.

[sormuras]

Regarding the failing org.junit.AutomaticModuleNameTests -- under the assumption, that junit-platform-engine-test is not intended to be used as a JPMS module, you exclude that sub-project from the check. Just add a filter here: https://github.com/junit-team/junit5/blob/master/platform-tests/src/test/java/org/junit/AutomaticModuleNameTests.java#L70

Or will s/o write requires org.junit.platform.engine.test in a module-info.java descriptor?

[dotCipher]

@sormuras I would not be opposed to having it JPMS compatible, but I haven't built much on java9. What would be involved with setting up the module-info.java ? Where would it go in junit5's project hierarchy?

I can filter out for now though.

[sormuras]

I'd say, filter it out for now. When 5.2 is released, I will revamp ``

Regarding the name of the proposed sub-project: the simple package test doesn't feel right. How about tck as in Technology Compatibility Kit? Or long: junit-platform-engine-tck, perhaps drop even the engine part would make sense, thus junit-platform-tck? With the package name adopted accordingly.

[dotCipher]

I went ahead with that refactor, let me know if it looks right. I will follow up with a commit to add javadocs + any relevant user guide docs as well

[sbrannen]

Regarding the name of the proposed sub-project: the simple package test doesn't feel right. How about tck as in Technology Compatibility Kit? Or long: junit-platform-engine-tck, perhaps drop even the engine part would make sense, thus junit-platform-tck? With the package name adopted accordingly.

It is not a TCK: it does not contain any tests at all.

Rather, it is a module for writing tests.

I currently think junit-platform-engine-test is a decent name: it is testing support for junit-platform-engine.

FWIW, Spring's testing support is published as spring-test. 😉

But I of course welcome brainstorming for better names.

[sbrannen]

If we don't want to limit the focus to testing support for engines, we could consider having a junit-platform-test (or junit-platform-test-support) module that may someday support testing of TestExecutionListener implementations, etc.

[dotCipher]

Fair enough, I can switch the naming it junit-platform-test to give potential room for growth. That sound reasonable?

[dotCipher]

I seem to be getting this warning though:

> Task :junit-jupiter-engine:junitPlatformTest
10:50:11.844 [main] WARN  org.junit.platform.launcher.core.DefaultLauncher - TestEngine with ID 'junit-jupiter' failed to discover tests
java.lang.NoClassDefFoundError: org/junit/platform/test/ExecutionRecorder
        at java.lang.Class.getDeclaredMethods0(Native Method) ~[?:?]
        at java.lang.Class.privateGetDeclaredMethods(Class.java:3139) ~[?:?]
        at java.lang.Class.getDeclaredMethods(Class.java:2266) ~[?:?]
        at org.junit.platform.commons.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:1120) ~[junit-platform-commons-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.commons.util.ReflectionUtils.findAllMethodsInHierarchy(ReflectionUtils.java:1061) ~[junit-platform-commons-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.commons.util.ReflectionUtils.getSuperclassMethods(ReflectionUtils.java:1276) ~[junit-platform-commons-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.commons.util.ReflectionUtils.findAllMethodsInHierarchy(ReflectionUtils.java:1064) ~[junit-platform-commons-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.commons.util.ReflectionUtils.findMethods(ReflectionUtils.java:1045) ~[junit-platform-commons-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.commons.util.ReflectionUtils.findMethods(ReflectionUtils.java:1031) ~[junit-platform-commons-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.jupiter.engine.discovery.JavaElementsResolver.resolveContainedMethods(JavaElementsResolver.java:189) ~[main/:?]
        at org.junit.jupiter.engine.discovery.JavaElementsResolver.resolveChildren(JavaElementsResolver.java:177) ~[main/:?]
        at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
        at org.junit.jupiter.engine.discovery.JavaElementsResolver.resolveClass(JavaElementsResolver.java:61) ~[main/:?]
        at java.util.ArrayList.forEach(ArrayList.java:1380) ~[?:?]
        at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1081) ~[?:?]
        at org.junit.jupiter.engine.discovery.DiscoverySelectorResolver.lambda$resolve$0(DiscoverySelectorResolver.java:59) ~[main/:?]
        at java.util.ArrayList.forEach(ArrayList.java:1380) ~[?:?]
        at org.junit.jupiter.engine.discovery.DiscoverySelectorResolver.resolve(DiscoverySelectorResolver.java:58) ~[main/:?]
        at org.junit.jupiter.engine.discovery.DiscoverySelectorResolver.resolveSelectors(DiscoverySelectorResolver.java:50) ~[main/:?]
        at org.junit.jupiter.engine.JupiterTestEngine.discover(JupiterTestEngine.java:61) ~[main/:?]
        at org.junit.platform.launcher.core.DefaultLauncher.discoverEngineRoot(DefaultLauncher.java:130) [junit-platform-launcher-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.launcher.core.DefaultLauncher.discoverRoot(DefaultLauncher.java:117) [junit-platform-launcher-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:90) [junit-platform-launcher-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.console.tasks.ConsoleTestExecutor.executeTests(ConsoleTestExecutor.java:65) [junit-platform-console-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.console.tasks.ConsoleTestExecutor.lambda$execute$0(ConsoleTestExecutor.java:57) [junit-platform-console-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.console.tasks.CustomContextClassLoaderExecutor.invoke(CustomContextClassLoaderExecutor.java:33) [junit-platform-console-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.console.tasks.ConsoleTestExecutor.execute(ConsoleTestExecutor.java:57) [junit-platform-console-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.console.ConsoleLauncher.executeTests(ConsoleLauncher.java:84) [junit-platform-console-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.console.ConsoleLauncher.execute(ConsoleLauncher.java:74) [junit-platform-console-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.console.ConsoleLauncher.execute(ConsoleLauncher.java:47) [junit-platform-console-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
        at org.junit.platform.console.ConsoleLauncher.main(ConsoleLauncher.java:39) [junit-platform-console-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
Caused by: java.lang.ClassNotFoundException: org.junit.platform.test.ExecutionRecorder
        at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582) ~[?:?]
        at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:496) ~[?:?]

Do I need to include the junit-platform-test as a test runtime dep for junit-jupiter-engine since it can't find it?

[dotCipher]

Opened up the refactor under a different branch name in PR #1392. Closing this one.