Parameterized test silently fails when arguments are incorrectly defined

[akutschera]

Given the following test

    @TestWithCases
    @TestCase(str1 = "foo", bool1 = true)
    @TestCase(str1 = "foo", bool1 = true)
    void xxx(String str, boolean val) {
        assertThat(val).isTrue();
    }

The test fails with

org.opentest4j.AssertionFailedError: 
Expecting value to be true but was false
Expected :true
Actual   :false

because there is nothing supplied for boolean val. The @TestCase annotation defined str1 and bool1 while it should have defined str1 and bool2, which leads to val having a default value of false.

What I would hope for is a message telling me something like 'cannot define first argument as string and boolean at the same timeornothing defined for second argumentor anything instead of just ignoring thebool1`.
The docs say that I cannot do it like this (which is good).
What I would like to discuss is if we should catch this mis-definition and tell the user about it instead of relying on the user to figure out what they mis-defined.

[Nylle]

Oh definitely! I'd love this to be an error, before executing the test - just like with customised fields that don't exist.

[Nylle]

Oh boy, that's a tricky one.

The problem is, that any annotation-method can only ever have a primitive type (including String and Class<T>), which is very unfortunate to begin with, because it massively limits the usefulness of TestCases.
On top of that, primitive types cannot be null, naturally, so they always need to have a default-value. This means, I don't know, how to detect whether one of the methods (str1() or bool2()) returns a set value or the default.
In fact, every TestCase, contains 6 default-values for all supported types, of which the ones specified in the annotation's parameters are overwritten.

In your test, when injecting the first argument, it will reflect the type (String) and execute the str1()-method on TestCase and get the configured value while for the second test-argument it will execute the TestCase::bool2()-method and receive its default-value.

The only hook on validation that I can see right now, would be finding out which arguments are actually configured in @TestCase(...) and so far I was not able to reflect that.

[jk-idealo]

I opened a PR for a fix. However I'm not sure, the error message is clearly describing the problem:

Duplicate customisation for test-method arguments at position 1, 2, 3, 4, 6

Any ideas for improvements?

[akutschera]

Maybe. When I change the test above to

    @TestCase(bool1 = true)
    @TestCase(bool1 = true)
    void xxx(String str, boolean val) {
        assertThat(val).isTrue();
    }

it will still fail with the PR as it is. I don't know how you can catch that when just validating the annotation. If you move the check to where you actually supply the values to the test parameters, this will probably work.
As a bonus, you could then construct an error message like
Duplicate customisation for test-method arguments at position 1, want String, also got boolean which would be a bit more verbose.
I would exit the checks on the first error. If you have multiple test cases that are mis-configured then you will have to re-run the tests multiple times until you fix them all.

[Nylle]

Good idea, I'll check this out.

[Nylle]

I completely screwed the rebase, so I re-created the branch and opened another PR.

[jk-idealo]

Fixed in version 2.9.7.