Cannot run individual instrumented tests with method parameters

[matejdro]

When attempting to run individual instrumented method that has parameters, IDE will just error out with "No tests found"

Example such test:

android-junit5/instrumentation/sample/src/androidTest/kotlin/de/mannodermaus/sample/ActivityOneTest.kt

Line 30 in a2f9034

fun testExample(scenario: ActivityScenario<ActivityOne>) {

[ktkopone]

I'm not familiar with the syntax being used in that example file to have parameters in an @Test, but if you're referring to running an individual @ParameterizedTest, I cannot reproduce your issue. I'm able to successfully run individual parameterized instrumented tests via either command line or Android Studio using version 1.2.0 of this repo.

[matejdro]

Here is example test:

class TestClass {
    @Test
    internal fun testMethod(@TempDir tempDir: File) {

    }
}

I've also noticed that I cannot reproduce it every time.

[mannodermaus]

As mentioned in the referenced issue, ADB and/or Android Studio seem to discard method signatures with parameters when submitting singular @Test runs to the instrumentation. The custom JUnit 5 runner is created properly, but its run() method is never invoked by the instrumentation. Looks like we'll have to sit tight until support for @Test methods with parameters is added.

Edit: Another thought I'd like to leave for further reference: The fact that @ParameterizedTest does work as pointed out by @ktkopone makes me think that somewhere in the Android instrumentation source code, test annotations are only checked by their simple name (i.e. Test as opposed to org.junit.jupiter.api.Test), then discarded when the signature has parameters because it only expects JUnit 4 at that point.

[mannodermaus]

I was able to conclude a deep and long investigation (don't worry though, it didn't take 1.5 years!). My previous assumption turned out to be semi-correct: The instrumentation discards any test method with parameters because its name doesn't match AndroidX's internal method filter. It works for parameterized tests because we happen to trigger an exemption from a different piece of the AndroidX code, wherein numerical suffixes are understood by the filters. This means that parameterizedTestMethod[0] is understood, but testMethod[String,String] is not.

I'll try to tackle this for the upcoming 1.3.0 release of the instrumentation runner library. A note for my future self: TestRequestBuilder$MethodFilter#evaluateTest() needs to match the test method name exactly, including round brackets and param types!

---

Research results

Before

The problem is seen in the first block, "Test with parameter". The final log says that the BlankRunner is used to run the suite, which is essentially a no-op as all tests were filtered out. The previous line highlights the difference between the expected name (testUsingMethodParameter(ActivityScenario)) and the actual one stored by AndroidX (testUsingMethodParameter).

// @Test with parameter
2021-07-27 20:52:56.193 13485-13511/de.mannodermaus.junit5.test I/System.out: RUNNER CREATED!! class de.mannodermaus.junit5.KotlinInstrumentationTests /// AndroidJUnit5RunnerParams(selectors=[MethodSelector [className = 'de.mannodermaus.junit5.KotlinInstrumentationTests', methodName = 'testUsingMethodParameter', methodParameterTypes = 'androidx.test.core.app.ActivityScenario']], filters=[], environmentVariables={}, systemProperties={}, configurationParameters={})
2021-07-27 20:52:56.557 13485-13511/de.mannodermaus.junit5.test I/System.out: My Runner: class androidx.test.internal.runner.TestRequestBuilder$ExtendedSuite
2021-07-27 20:52:56.743 13485-13511/de.mannodermaus.junit5.test I/System.out: evaluateTest() testUsingMethodParameter(ActivityScenario)... included=[testUsingMethodParameter]
2021-07-27 20:52:56.833 13485-13511/de.mannodermaus.junit5.test I/System.out: Finally, the runner used: androidx.test.internal.runner.TestRequestBuilder$BlankRunner@5dcf41c

// @Test without parameter
2021-07-27 20:53:41.158 13618-13644/de.mannodermaus.junit5.test I/System.out: RUNNER CREATED!! class de.mannodermaus.junit5.KotlinInstrumentationTests /// AndroidJUnit5RunnerParams(selectors=[MethodSelector [className = 'de.mannodermaus.junit5.KotlinInstrumentationTests', methodName = 'testUsingGetScenario', methodParameterTypes = '']], filters=[], environmentVariables={}, systemProperties={}, configurationParameters={})
2021-07-27 20:53:41.504 13618-13644/de.mannodermaus.junit5.test I/System.out: My Runner: class androidx.test.internal.runner.TestRequestBuilder$ExtendedSuite
2021-07-27 20:53:41.669 13618-13644/de.mannodermaus.junit5.test I/System.out: evaluateTest() testUsingGetScenario... included=[testUsingGetScenario]
2021-07-27 20:53:41.741 13618-13644/de.mannodermaus.junit5.test I/System.out: Finally, the runner used: androidx.test.internal.runner.TestRequestBuilder$ExtendedSuite@5dcf41c
2021-07-27 20:53:41.757 13618-13644/de.mannodermaus.junit5.test I/System.out: RUNNER RUN!!!

// @ParameterizedTest
2021-07-27 20:54:21.158 13787-13813/de.mannodermaus.junit5.test I/System.out: RUNNER CREATED!! class de.mannodermaus.junit5.KotlinInstrumentationTests /// AndroidJUnit5RunnerParams(selectors=[MethodSelector [className = 'de.mannodermaus.junit5.KotlinInstrumentationTests', methodName = 'kotlinTestWithParameters', methodParameterTypes = 'int']], filters=[], environmentVariables={}, systemProperties={}, configurationParameters={})
2021-07-27 20:54:21.475 13787-13813/de.mannodermaus.junit5.test I/System.out: My Runner: class androidx.test.internal.runner.TestRequestBuilder$ExtendedSuite
2021-07-27 20:54:21.583 13787-13813/de.mannodermaus.junit5.test I/System.out: Finally, the runner used: androidx.test.internal.runner.TestRequestBuilder$ExtendedSuite@5dcf41c
2021-07-27 20:54:21.605 13787-13813/de.mannodermaus.junit5.test I/System.out: RUNNER RUN!!!

// @RepeatedTest
2021-07-27 20:55:50.056 14009-14035/de.mannodermaus.junit5.test I/System.out: RUNNER CREATED!! class de.mannodermaus.junit5.KotlinInstrumentationTests /// AndroidJUnit5RunnerParams(selectors=[MethodSelector [className = 'de.mannodermaus.junit5.KotlinInstrumentationTests', methodName = 'kotlinRepeatedTest', methodParameterTypes = 'org.junit.jupiter.api.RepetitionInfo']], filters=[], environmentVariables={}, systemProperties={}, configurationParameters={})
2021-07-27 20:55:50.385 14009-14035/de.mannodermaus.junit5.test I/System.out: My Runner: class androidx.test.internal.runner.TestRequestBuilder$ExtendedSuite
2021-07-27 20:55:50.474 14009-14035/de.mannodermaus.junit5.test I/System.out: Finally, the runner used: androidx.test.internal.runner.TestRequestBuilder$ExtendedSuite@80d0dfa
2021-07-27 20:55:50.504 14009-14035/de.mannodermaus.junit5.test I/System.out: RUNNER RUN!!!

// @TestFactory
2021-07-27 20:57:02.217 14209-14235/de.mannodermaus.junit5.test I/System.out: RUNNER CREATED!! class de.mannodermaus.junit5.KotlinInstrumentationTests /// AndroidJUnit5RunnerParams(selectors=[MethodSelector [className = 'de.mannodermaus.junit5.KotlinInstrumentationTests', methodName = 'kotlinTestFactory', methodParameterTypes = '']], filters=[], environmentVariables={}, systemProperties={}, configurationParameters={})
2021-07-27 20:57:02.778 14209-14235/de.mannodermaus.junit5.test I/System.out: My Runner: class androidx.test.internal.runner.TestRequestBuilder$ExtendedSuite
2021-07-27 20:57:02.928 14209-14235/de.mannodermaus.junit5.test I/System.out: Finally, the runner used: androidx.test.internal.runner.TestRequestBuilder$ExtendedSuite@194a1a1
2021-07-27 20:57:02.959 14209-14235/de.mannodermaus.junit5.test I/System.out: RUNNER RUN!!!

[mannodermaus]

Released in instrumentation 1.3.0! Please note that the new version requires JUnit 5.8.0