"Expected N+1 tests, received N" caused by fun with @TestTemplate

[mgroth0]

I was getting "Expected N+1 tests, received N" errors, and at first it was very difficult to find the cause. After a thorough investigation I found that:

• The "Expected N+1 tests, received N" error was only happening for android instrumented Junit5 tests.
• Did not occur for regular JVM Junit5 tests or android Junit 5 unit tests
• Only occured when a certain top-level function was on the classpath with a signature like:

@TestTemplate fun someFunction() {}

If I comment out @TestTemplate above, the problem goes away.

For context, note that someFunction above was just some scrap code I through somewhere when I was messing around and forget to get rid of. It is located in a common test dependency that is shared by JVM desktop, android unit tests, and android instrumented tests. It never caused any issue until I set up android-junit5 for my instrumented tests.

[mannodermaus]

Hey, thanks for reporting. First of all, I'd like to ask what version of JUnit 5, the android-junit5 plugin and the instrumentation libraries you are using in this project.

I'm trying to repro your case but test template functions are discovered and executed just fine. For completeness sake, did your scrap function only have the @TestTemplate on it, or did you omit something for brevity? The thing is, a test template is supposed to be used with some other annotation that provides context to what the template is generating.

// Don't
@TestTemplate
fun myTemplate1() {}

// Do
@TestTemplate
@ExtendWith(SomeTemplateContextProvider::class)
fun myTemplate2(param: SomeTemplateContext) {}

If you are missing that context provider, maybe this is the reason why you were seeing the miscount of the test engine? I'm not getting a miscount on the latest dev version of the libraries, even when I purposefully misconfigure the template declaration, so it's possible that I unknowingly fixed something in the meantime.

[mgroth0]

Hey, I just got a reproducer.

I know this is an unusual condition that would trigger the bug, but I think it is important because the error messafge is so uninformative that it could really stump people. Maybe it can't be totally fixed, but if you at least understand what is happening maybe you can somehow ensure that it fails in a better way so people can quickly correct it?

build.gradle.kts

import org.gradle.api.JavaVersion.VERSION_17

plugins {
    kotlin("multiplatform") version("2.0.0-Beta5")
    id("com.android.library") version "8.2.0"
    id("de.mannodermaus.android-junit5") version "1.10.0.0"
}

kotlin {
    jvmToolchain(17)
}


repositories {
    mavenCentral()
    google()
}


kotlin {
    androidTarget()
}
android {
    namespace = "bug"
    compileSdk = 34
    compileOptions {
        sourceCompatibility = VERSION_17
        targetCompatibility = VERSION_17
    }
    defaultConfig {
        minSdk = 33
        testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
        testInstrumentationRunnerArguments["runnerBuilder"] ="de.mannodermaus.junit5.AndroidJUnit5Builder"
    }

}

dependencies {
    val cfg = "androidInstrumentedTestImplementation"
    add(cfg,"org.junit.jupiter:junit-jupiter-api:5.10.2")
    add(cfg,"de.mannodermaus.junit5:android-test-core:1.4.0")
    add(cfg,"de.mannodermaus.junit5:android-test-runner:1.4.0")
}

settings.gradle.kts

pluginManagement {
    repositories {
        mavenCentral()
        gradlePluginPortal()
        google()
    }
}

gradle.properties

android.useAndroidX=true

src/androidInstrumentedTest/kotlin/test/test.kt

package test

import org.junit.jupiter.api.Test
import org.junit.jupiter.api.TestTemplate


class Tests {
    @Test
    fun test() {
        println("hello test")
    }
}


@TestTemplate
private fun test() {}

[mannodermaus]

Thank you for the thorough reproducer, this was super helpful in getting to the bottom of this. Your usage of Kotlin Multiplatform had me worried for a bit, but it turns out that it doesn't have any say in the issue appearing. After a few hours of tracing and digging, I was finally able to pinpoint the culprit and fix it. Some new unit tests make sure that top-level functions won't be prematurely considered by the JUnit 5 builder going forward, preventing a mismatch between what we report to the Android instrumentation at the start, and what JUnit 5 ends up actually executing.

This will be available in the upcoming 1.5.0 release of the instrumentation libraries (the runner artifact, to be precise). If you need the fix right now, you can use the 1.5.0-SNAPSHOT instead. Cheers!

[mgroth0]

Nice bug squashing! As small as this was it's always nice to have testing frameworks you can really depend on :)