Extend support for JUnit 5 via the new roborazzi-junit5 artifact

[mannodermaus]

This PR introduces roborazzi-junit5, a newly proposed artifact for extending the capabilities of this fantastic library with the newest version of JUnit. It refactors how Roborazzi's file name generation works and adds a second detection algorithm specifically for tests using the JUnit Jupiter API from JUnit 5.

Background

Historically, Robolectric-based tests have not been supported by JUnit 5 because of incompatibilities with the way it injects custom class loading into the execution environment. However, very recently there have been some fantastic achievements in this space, finally allowing JUnit 5 to deal with Robolectric tests via a third-party extension. The hope is to integrate this extension into the mainline Robolectric library in the future. With this extension in place, it's already possible to use the current Roborazzi version for ordinary test cases, but only if you specify the file path manually for every usage of captureRoboImage.

Motivation

The default behavior of captureRoboImage delegates the final file name of the captured image to the DefaultFileNameGenerator. This class can infer the file name by looking at the stacktrace of the Robolectric thread and finding the element annotated with @org.junit.Test. There is also some basic detection for the JUnit 5 annotation (@org.junit.jupiter.api.Test), but the check doesn't cover all of the possible cases and fails when used together with the extension I mentioned earlier. Consider the following test class as an example:

@GraphicsMode(NATIVE)
@ExtendWith(RobolectricExtension::class)
class MyJUnit5RoborazziTest {
  @ParameterizedTest
  @ValueSource(strings = ["first", "second"])
  fun myTest(value: String) {
    // ...
    onView(isRoot()).captureRoboImage() // Will fail: "Roborazzi can't find method of test"
  }
}

Approach

This PR refactors the stacktrace detection code and makes it an implementation of the new TestNameExtractionStrategy interface inside of roborazzi-core. By default, Roborazzi will always use this implementation for generating file names, just like before. Additionally, it can detect if roborazzi-junit5 is on the classpath, and if this is the case, it also adds the new JUnit5TestNameExtractionStrategy to itself. You see, the issue with JUnit 5 stacktraces is that they don't always contain the exact test method in them, so you cannot find them by their annotation. Furthermore, there are a whole bunch of annotations that generate tests at runtime, and those cases don't have any annotations in the trace and therefore cannot be detected with the current way:

• @ParameterizedTest
• @RepeatedTest
• @TestFactory
• @TestTemplate

The extraction strategy for JUnit 5 that I added to roborazzi-junit5 is tailored to detecting the names of these tests. It reads the currently executed test method from a shared storage called CurrentTestInfo, which is updated from the other side by a JUnit 5 extension that keeps the storage up-to-date at all times. Since both sides are attached to different class loaders, there is a fair bit of ugly trickery involved to send this information across their boundaries. 😵 Essentially, the extraction strategy has to look up the shared CurrentTestInfo via reflection, otherwise it cannot see the other class loader's static data. I'm using lazy references as much as possible to minimize the runtime performance impact of this.

The following diagram illustrates how the classes in the new artifact work together:

Setup

With this new artifact, consumers can use Roborazzi in their Robolectric tests with JUnit 5. The proposed setup would look like this:

Step 1: Add the new dependency next to the main Roborazzi library

 dependencies {
   testImplementation("org.junit.jupiter:junit-jupiter-api:x.xx.x")
   testRuntimeOnly("org.junit.jupiter:junit-jupiter-engine:x.xx.x")
   testImplementation("io.github.takahirom.roborazzi:roborazzi:x.xx.x")
+  testImplementation("io.github.takahirom.roborazzi:roborazzi-junit5:x.xx.x")
 }

Step 2: Enable Roborazzi's JUnit 5 extension

Any of the following options will work, and only one of them needs to be followed. I personally prefer option C.

Option A: Add the Roborazzi extension to each test class

 // in MyJUnit5RoborazziTest.kt
 @GraphicsMode(NATIVE)
- @ExtendWith(RobolectricExtension::class)
+ @ExtendWith(RobolectricExtension::class, RoborazziExtension::class)
 class MyJUnit5RoborazziTest {
   @ParameterizedTest
   @ValueSource(strings = ["first", "second"])
   fun myTest(value: String) {
     // ...
     onView(isRoot()).captureRoboImage()
   }
 }

Option B: Enable autodetection of extensions globally via properties file

 // in src/test/junit-platform.properties
+ junit.jupiter.extensions.autodetection.enabled=true

 (no change to MyJUnit5RoborazziTest necessary)

Option C: Enable autodetection of extensions via android-junit5's Gradle DSL

 // in app/build.gradle.kts
+ junitPlatform {
+   configurationParameter("junit.jupiter.extensions.autodetection.enabled", "true")
+ }

 (no change to MyJUnit5RoborazziTest necessary)

Conclusion

Apologies for dropping this wall of text unprompted! Please let me know your thoughts. 🙇‍♂️ This PR may be incomplete, as I have not yet looked into the publishing part of this new artifact. If there are any other files that need updating, let me know.

[takahirom]

Thanks for your contribution.
Overall, I want to introduce these changes, and I was able to understand them from the pull request description. I do have a few questions, but they are minor.
Could you take a look at the CI failure?

[takahirom]

Question: Will the RoborazziExtension be automatically installed in all users' projects by this file for every test? Can users disable it?

[mannodermaus]

Yes, but only if they enable the automatic detection of extensions, either via Option B or C listed above. If they don't do either, then the extension would not be installed and they would have to use @ExtendWith(RoborazziExtension::class) for every test

[takahirom]

I wasn't aware that we needed to use ThreadLocal. I'm a bit concerned that we are currently using object for RoborazziContext, and if we switch to ThreadLocal, it might break some tests that use a different thread. I think this issue isn't directly related to this PR, but we should consider it later.

[mannodermaus]

It's a bit of a premature optimization at this point, because we haven't figured out parallel test execution for Robolectric yet (and I suppose that limitation extends to Roborazzi). But in the future, it's theoretically possible for multiple tests to run concurrently on different threads, and every test should be able to store its own TestInfo here

[takahirom]

It may be challenging, but could we implement tests for ExecutionMode.CONCURRENT? 👀

[mannodermaus]

Yeah it should work, let me think about this!

[mannodermaus]

I added a concurrent unit test battery for CurrentTestInfo in 4f07c4b

[takahirom]

👍

[mannodermaus]

Thank you for the swift review! 🙇‍♂️ I will address the CI failure shortly and consider extending the unit tests.

[mannodermaus]

@takahirom お待たせしました。We should be good to go with another run of the workflows. I reinstated the basic JUnit 5 detection of the stack trace strategy in the latest commit 0d8c86e, since apparently Kotlin Test defaults to using JUnit 5 behind the scenes for Compose Desktop and Multiplatform environments in some cases. With that, I was able to run all tests locally fine, including the ones in the sample projects. 🤞

[takahirom]

Looks great. I should have mentioned this earlier, but I would like to add some tests and documentation.

• I think we can add roborazzi-junit5 as a dependency for sample-compose-desktop-jvm. This way, we can have integration tests for sample-compose-desktop-jvm, as it would be the default choice for Compose Desktop's tests.
• Can we have a sample app for JUnit5 Robolectric tests to help me notice bugs with JUnit5?
• Could we add documentation (/docs) for roborazzi-junit5?

[mannodermaus]

Absolutely, sounds like a solid plan. I'll need a bit of time for this, but I'll let you know once done.

[mannodermaus]

@takahirom I'm thinking about the RoborazziRule and whether or not to make a similar functionality available with the JUnit 5 integration as well (since you wouldn't be able to use the JUnit 4-based rule with the newer API). Would this be something you are interested in?

[github-actions]

Snapshot diff report

File name	Image
MainJvmTest.testFact ory_4_compare.png
MainJvmTest.repeated Test_4_compare.png
MainJvmTest.paramete rizedTest_3_compare. png
MainJvmTest.paramete rizedTest_compare.pn g
MainJvmTest.repeated Test_compare.png
MainJvmTest.testFact ory_compare.png
MainJvmTest.repeated Test_3_compare.png
MainJvmTest.testFact ory_3_compare.png
MainJvmTest.testFact ory_2_compare.png
MainJvmTest.paramete rizedTest_2_compare. png
MainJvmTest.repeated Test_2_compare.png
MainJvmTest.paramete rizedTest_4_compare. png
com.github.takahirom .roborazzi.sample.JU nit5ManualTest.captu reRoboImageSample_2_ compare.png
com.github.takahirom .roborazzi.sample.JU nit5ManualTest.captu reRoboImageSample_co mpare.png
com.github.takahirom .roborazzi.sample.JU nit5ManualTest.testF actory_2_compare.png
com.github.takahirom .roborazzi.sample.JU nit5ManualTest.testF actory_compare.png
com.github.takahirom .roborazzi.sample.JU nit5ManualTest.param eterizedTest_2_compa re.png
com.github.takahirom .roborazzi.sample.JU nit5ManualTest.param eterizedTest_compare .png
com.github.takahirom .roborazzi.sample.JU nit5ManualTest.repea tedTest_compare.png
com.github.takahirom .roborazzi.sample.JU nit5ManualTest.repea tedTest_2_compare.pn g

[takahirom]

@mannodermaus
I think if we could have a mechanism similar to JUnit4 rules, that would be great, but I'm fine with the current implementation for now. We can add that mechanism to JUnit5 later.

[mannodermaus]

Seems like we have a compatibility issue because the Robolectric JUnit 5 extension is built against Java 17, while Roborazzi uses the more conversative Java 11. 🤔 AGP has had 17 as the minimum version since 8.0, would this be something you'd be willing to raise? Roborazzi could keep its targetCompatibility at 11, but use a higher-level toolchain for compilation.

[mannodermaus]

@takahirom I wrote a doc section on JUnit 5 in the latest commit (first time using Writerside; pretty neat!) and also updated the Robolectric extension to the latest version. It's now targeting Java 11, so there shouldn't be any compatibility issues - let's just hope that the integration tests finally work.

[takahirom]

I think this is easy-to-understand documentation! Thanks!