Dataprovider caches results even for dynamically created data / results (e.g. using test method as parameter)

[weiro-9-w7]

Hello , would you help to check the issue.
I write a sample in https://github.com/weiro-9-w7/junit-data-provider
when you DataProviderExternelFileTest testcase, FrameworkMethod get annotation is the other method annotation.

@RunWith(DataProviderRunner.class)
public class DataProviderExternelFileTest {

@Test
@UseDataProvider(value = "loadFromExternalFile", location = LoadFromExternalFile.class)
@ExternalFile(format = ExternalFile.Format.JSON, value = "persons.json", clazz=Person.class, isArray=true)
public void should_return_array_when_is_array_is_true(List<Person> person) {
    // Expect:
    assertEquals(person.get(0).getName(), "zhangsan");
    assertEquals(person.get(0).getAge(), 18);
}

@Test
@UseDataProvider(value = "loadFromExternalFile", location = LoadFromExternalFile.class)
@ExternalFile(format = ExternalFile.Format.JSON, value = "person.json", clazz=Person.class)
public void should_return_object_when_is_array_is_false(Person person) {
    // Expect:
    assertEquals(person.getName(), "zhangsan");
    assertEquals(person.getAge(), 18);
}

}

public class LoadFromExternalFile {
@dataProvider
public static Object[][] loadFromExternalFile(FrameworkMethod testMethod) throws IOException {
ExternalFile externalFile = testMethod.getAnnotation(ExternalFile.class);
String testDataFile = externalFile.value();
File file = new File(LoadFromExternalFile.class.getResource("/").getPath() + testDataFile);
String content = FileUtils.readFileToString(file, "utf8");
if(externalFile.isArray()){
return new Object[][]{{JSON.parseArray(content, externalFile.clazz())}};
}else{
return new Object[][]{{JSON.parseObject(content, externalFile.clazz())}};
}
}
}

when you run should_return_array_when_is_array_is_true method, but ExternalFile externalFile = testMethod.getAnnotation(ExternalFile.class); externalFile is should_return_object_when_is_array_is_false

[weiro-9-w7]

com.tngtech.junit.dataprovider junit4-dataprovider 2.0 test

[aaschmid]

Hi @weiro-9-w7,

I will have a look as soon as I can. How urgent is the topic for you?

Cheers,
Andreas

[weiro-9-w7]

Hi @aaschmid ,
I'm glad you can give feedback to me in time. We prepare to use Junit-data-provider on the legacy systems . The function of different methods to use different annoation data is very important to us. So we hope to help solve it as soon as possible.

[aaschmid]

Hi @weiro-9-w7,

to be honest, I don't get your problem. Everything is working fine on my machine, see this screenshot:

Test should_return_array_when_is_array_is_true got an array and the other didn't get one, as expected.

Is there another issue, I haven't understood yet?

[weiro-9-w7]

Hi @aaschmid ,
yes, you can't get any problems because I move one method to other class.
When one class have two methods and use annotation, issues will happen.
Would you run test for DataProviderExternelFileMixTest. you will found problem.

[weiro-9-w7]

[weiro-9-w7]

[aaschmid]

Hi @weiro-9-w7,

problem is that the dataprovider results are cached per test class such that your dataprovider method is only called once for the first test case in this class. As JUnit4 always evaluates all test cases before executing it, it does not matter if you only execute one test case or all test cases in that class.

This caching mechanism was introduced by feature request, see #93. The caching, though, only occurs per test class in JUnit4. In JUnit5 the caching even occurs on per test run ...

I guess I should distinguish if a dataprovider takes a further argument (do not cache result) or not (cache result). Another possibility could be another setting option on @DataProvider annotation.

I will try to provide a fix with a new version on the weekend.

Cheers,
Andreas

[weiro-9-w7]

@aaschmid
thanks

[aaschmid]

Hi @weiro-9-w7,
I fixed the issue by introducing a cache option on @DataProvider, see commit 143948f (or above). I just need to stabilize the test cases (very fragile currently). You can test it manually by cloning the repo and creating your own version locally.

Any feedback welcome.
Cheers,
Andreas

[aaschmid]

Works with new version v2.2, see

[weiro-9-w7]

Hi @aaschmid , verified, thanks!

[shiv20934]

Getting the below issue after using v2.2
java.lang.ArrayStoreException: sun.reflect.annotation.TypeNotPresentExceptionProxy
at sun.reflect.annotation.AnnotationParser.parseClassArray(Unknown Source)
at sun.reflect.annotation.AnnotationParser.parseArray(Unknown Source)
at sun.reflect.annotation.AnnotationParser.parseMemberValue(Unknown Source)
at java.lang.reflect.Method.getDefaultValue(Unknown Source)
at sun.reflect.annotation.AnnotationType.(Unknown Source)
at sun.reflect.annotation.AnnotationType.getInstance(Unknown Source)
at sun.reflect.annotation.AnnotationParser.parseAnnotation2(Unknown Source)
at sun.reflect.annotation.AnnotationParser.parseAnnotations2(Unknown Source)
at sun.reflect.annotation.AnnotationParser.parseAnnotations(Unknown Source)
at java.lang.reflect.Executable.declaredAnnotations(Unknown Source)
at java.lang.reflect.Executable.declaredAnnotations(Unknown Source)
at java.lang.reflect.Executable.getDeclaredAnnotations(Unknown Source)
at java.lang.reflect.Method.getDeclaredAnnotations(Unknown Source)
at java.lang.reflect.AccessibleObject.getAnnotations(Unknown Source)
at org.junit.runners.model.FrameworkMethod.getAnnotations(FrameworkMethod.java:189)
at org.junit.runners.model.TestClass.addToAnnotationLists(TestClass.java:58)
at org.junit.runners.model.TestClass.(TestClass.java:46)
at org.junit.runners.ParentRunner.(ParentRunner.java:75)
at org.junit.runners.BlockJUnit4ClassRunner.(BlockJUnit4ClassRunner.java:57)
at com.tngtech.java.junit.dataprovider.DataProviderRunner.(DataProviderRunner.java:79)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:29)
at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:21)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader.createUnfilteredTest(JUnit4TestLoader.java:90)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader.createTest(JUnit4TestLoader.java:76)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader.loadTests(JUnit4TestLoader.java:49)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:526)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:770)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:464)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210)

[aaschmid]

@shiv20934: Can you provide more information and / or use v2.8 instead which might not have that issue anymore?