-
Notifications
You must be signed in to change notification settings - Fork 53
Conditional Test Execution
One of the most exciting features of JUnit 5 is the functionality provided by its Conditional Test Execution annotations. At runtime, JUnit Jupiter evaluates external properties in order to make a decision about whether or not an annotated test method should be executed. This allows developers to write specific tests that only run on certain Java versions or operating systems, or depend on certain environment variables or system properties, for example. This guide introduces those annotations in the context of the Android ecosystem.
Since version 1.2.0, the android-test-core library includes a couple of annotations that drive conditional test execution in scenarios tailored towards Android developers. The following sections introduce these annotations and their effect on the execution of instrumentation tests.
Skip or drive execution for manufacturer-specific test cases. This helps with testing bugs scoped towards a certain subset of devices. For example, you can run a test only on Samsung devices, or skip it if the test suite is executed on a Huawei device.
// Matches e.g. "Samsung", "SAMSUNG", "sAmSuNg" if ignoreCase == true (default),
// and "Samsung" only if ignoreCase == false
@EnabledOnManufacturer(value = "Samsung", ignoreCase = true)
@Test
fun testForSamsungDevicesOnly() {
}
// Matches e.g. "Google", "GOOGLE", "gOoGlE" if ignoreCase == true (default),
// and "Google" only if ignoreCase == false
@DisabledOnManufacturer(value = "Google", ignoreCase = true)
@Test
fun testForAllButGoogleDevices() {
}If one of these annotations causes a test to be skipped, you can find the following line in the Logcat output:
W/AndroidJUnit5: testForSamsungDevicesOnly is ignored. Disabled on Manufacturer: OnePlus
Skip or drive execution on specific API levels. This helps with testing apparent behavior on a certain range of OS versions. For example, you can run a test only on devices running Android P or newer, or skip it if the test suite is executed on a device older than Q.
// There is also an "until" parameter, indicating the highest API level where the test should be enabled
@EnabledOnSdkVersion(from = 28)
@Test
fun testRunningOnPieAndAbove() {
}
// There is also a "from" parameter, indicating the lowest API level where the test should be disabled
@DisabledOnSdkVersion(until = 28)
@Test
fun testNeverRunningOnPieAndBelow() {
}If one of these annotations causes a test to be skipped, you can find the following line in the Logcat output:
W/AndroidJUnit5: testRunningOnPieAndAbove is ignored. Disabled on API 27
Skip or drive execution based on specific BuildConfig fields. When defining certain features through flags located in the build config, or testing flavor-specific logic, these annotations may come in handy. It's important to note that while BuildConfig fields can be of arbitrary type, these annotations will convert them into strings before matching against a regular expression.
// This test will be executed if the field "MY_KEY" contains four characters, e.g. when specifying:
// android.defaultConfig.buildConfigField("String", "MY_KEY", "\"ABCD\"")
@EnabledIfBuildConfigValue(named = "MY_KEY", matches = "\\w{4}")
@Test
fun testRunningWhenMyKeyHasFourCharacters() {
}
// This test will be skipped if the field "MY_INT", converted to string, equals "100", e.g. when specifying:
// android.defaultConfig.buildConfigField("int", "MY_INT", "100")
@DisabledIfBuildConfigValue(named = "MY_INT", matches = "100")
@Test
fun testSkippingWhenMyIntIsOneHundred() {
}If one of these annotations causes a test to be skipped, you can find one of the following lines in the Logcat output:
testRunningWhenMyKeyHasFourCharacters is ignored. BuildConfig key [MY_KEY] with value [13370815] does not match regular expression [\\w{4}]
testSkippingWhenMyIntIsOneHundred is ignored. BuildConfig key [MY_VALUE] with value [100] matches regular expression [100]
testSomethingEntirelyDifferent is ignored. BuildConfig key [NON_EXISTENT_KEY] does not exist
The JUnit Jupiter API comes with its own set of annotations for conditional test execution. In the daily workflow of an Android developer, some of these yield more benefit than others. This section describes some considerations for using them efficiently.
Android instrumentation tests will likely run only on Android devices. Therefore, this annotation pair doesn't really have much benefit. In fact, an Android device will match the OS.LINUX value, causing tests annotated with @EnabledOnOs(OS.LINUX) to always run, and tests annotated with @DisabledOnOs(OS.LINUX) to always be skipped. Other values of the enum will never match.
These annotations look up the value of the java.version system property, as well as some internals of the java.lang.Runtime class. On Android, these values are unset, causing a reported JRE version of 0 at all times. Do not rely on these annotations for Android - if you want to restrict execution based on the system, @EnabledOnSdkVersion and @DisabledOnSdkVersion are a better choice.
It isn't possible out of the box to provide environment variables to the Android instrumentation, outside of tools like gcloud and Firebase Test Lab. The JUnit 5 runner provides a handy feature for advanced usage to developers, allowing them to utilize @EnabledIfEnvironmentVariable and @DisabledIfEnvironmentVariable just like with JVM tests. You can provide them as a comma-separated value to the test runner in your build.gradle setup, as shown below:
android {
defaultConfig {
// Right below where you specify the Test Runner & JUnit 5 itself...
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
testInstrumentationRunnerArgument("runnerBuilder", "de.mannodermaus.junit5.AndroidJUnit5Builder")
// ...you can provide this argument, too!
testInstrumentationRunnerArgument("environmentVariables", "KEY1=value1,KEY2=value2")
}
}The environmentVariables key is read in and processed by the instrumentation runtime. If you need to provide multiple variables, please make sure to provide this argument only once, and separate different arguments with commas.
Using the above setup, the following behavior works as described in the comments:
@EnabledIfEnvironmentVariable(named = "KEY1", matches = "value1")
@Test
fun testSomething() {
// This will be executed, because the variable exists with a matching value
}
@DisabledIfEnvironmentVariable(named = "KEY2", matches = "value2")
@Test
fun skipSomething() {
// This will not be executed, because the variable exists with a matching value
}Support for JUnit 5's system-property-based annotations is possible in a similar fashion to the environment variables.You can provide them as a comma-separated value to the test runner in your build.gradle setup, as shown below:
android {
defaultConfig {
// Right below where you specify the Test Runner & JUnit 5 itself...
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
testInstrumentationRunnerArgument("runnerBuilder", "de.mannodermaus.junit5.AndroidJUnit5Builder")
// ...you can provide this argument, too!
testInstrumentationRunnerArgument("systemProperties", "KEY1=value1,KEY2=value2")
}
}The systemProperties key is read in and processed by the instrumentation runtime. If you need to provide multiple variables, please make sure to provide this argument only once, and separate different arguments with commas.
Using the above setup, the following behavior works as described in the comments:
@EnabledIfSystemProperty(named = "KEY1", matches = "value1")
@Test
fun testSomething() {
// This will be executed, because the property exists with a matching value
}
@DisabledIfSystemProperty(named = "KEY2", matches = "value2")
@Test
fun skipSomething() {
// This will not be executed, because the variable exists with a matching value
}