Surefire Plugin should evaluate @DisplayName

[mab]

Overview

Feature request.

Currently the surefire plugin does not evaluate @DisplayName in the test reports. It would be great if Surefire would use @DisplayName where present.

Maven surfire output from the junit5-maven-consumer project:

Running com.example.project.FirstTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.115 sec <<< FAILURE! - in com.example.project.FirstTest
myFirstTest(TestInfo)  Time elapsed: 0.035 sec  <<< FAILURE!
org.opentest4j.AssertionFailedError: 1 + 1 should equal 2 ==> expected: <2> but was: <3>
        at com.example.project.FirstTest.myFirstTest(FirstTest.java:27)

In contrast the output from the IntelliJ plugin using the @DisplayName results in more readable test names.

Desired out put would be

Running FirstTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.115 sec <<< FAILURE! - in FirstTest
My 1st JUnit 5 test! 😎  Time elapsed: 0.035 sec  <<< FAILURE!
org.opentest4j.AssertionFailedError: 1 + 1 should equal 2 ==> expected: <2> but was: <3>
        at com.example.project.FirstTest.myFirstTest(FirstTest.java:27)

For parameterized Tests it would be great to have an output like in IntelliJ in the surefire plugin to. (DisplayName, Iteration and value of the argument causing a failed test)

Deliverables

• ...

[marcphilipp]

The rationale behind not using display names is that it would "break" the XML report generated by Surefire. Thus, we need a new reporting format and Surefire needs to adopt it before we can report display names (cf. #373).

[sbrannen]

+1 to what @marcphilipp said: it's unfortunately not possible to support all JUnit Platform reporting features in reporting systems that only support JUnit 4.

So, the entire world has to wait until there is a new reporting standard which supports custom display names, test sources, etc.

[luolong]

The rationale behind not using display names is that it would "break" the XML report generated by Surefire.

So how come Gradle reports are fine?

[marcphilipp]

So how come Gradle reports are fine?

Because our Gradle plugin uses ConsoleLauncher to write these XML reports.

[mibutec]

Same for Dynamic Tests

Reporting in CI environment

Error | testFactoryMethodName[1][1]
Error | testFactoryMethodName[1][4]
Error | testFactoryMethodName[1][6]
Error | testFactoryMethodName[1][7]
Error | testFactoryMethodName[1][8]

Could it be a solution to enode display names in a way they are compatible to surefire, i.e. My 1st JUnit 5 test! 😎 => My_1st_JUnit_5_test

[mibutec]

Is there any workaround?

[mibutec]

@marcphilipp @sbrannen

How about this? For now Dynamic Tests are hardly to use in CI...

[sbrannen]

Since JUnit 5's Surefire provider is currently being taken over by the Maven team, the JUnit Team is not actively working on Surefire related issues.

[dantran]

I filed an enhancement at https://issues.apache.org/jira/browse/SUREFIRE-1567

[k1w1m8]

per #1320

I was wondering what was happening in Surefire wrt JUnit5.

It appears there is a single committer on the project. They are obviously doing an heroic job there, but there is a limit to what one person can do. Therefore, tickets for fixing the reporting of new JUnit5 features are not being worked on.

https://issues.apache.org/jira/browse/SUREFIRE-1567
https://issues.apache.org/jira/browse/SUREFIRE-1546

IMHO, having JUnit5 without first class support for new features in Surefire is pretty pointless given the importance of Maven to the Java ecosystem.

Is this something that the JUnit5 team are concerned about? Is there a plan to address this issue?

I've never worked on Surefire, but I'm considering getting involved...

[sbrannen]

@k1w1m8, please see my response in #1320 (comment).

[k1w1m8]

Thanks @sbrannen.

[sormuras]

Closing this issue.

fup2 https://issues.apache.org/jira/projects/SUREFIRE/issues/SUREFIRE-1546

[sormuras]

https://issues.apache.org/jira/projects/SUREFIRE/issues/SUREFIRE-1546 was just closed -- the feature will available in Surefire 3.0.0-M4, soon.

[WalidBounouar]

Hi, apologies if this is not the correct place to ask, but is there some way to follow the status of 3.0.0-M4 (expected release)?
Thank you!

[marcphilipp]

You can ask on the Maven User List.

[Tibor17]

@WalidBounouar the version 3.0.0-M4 was released yesterday. You can always ask me, see my email in GitHub, or you can subscribe to the Maven user-mailing list and ask the user questions in public. If you are in the Apache, you can find us in the Slack.

[arun-mano]

Hi Tibor,
I was working with JUnit 5, creating dynamic tests, when I run the build using maven,
The XML file under /surefire-reports has the information as expected.

But in the Console logs still lists with indexes, and the test case Displayname is not printed.

I have the below Plugin config:
<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>3.0.0-M5</version> <dependencies> <dependency> <groupId>org.junit.jupiter</groupId> <artifactId>junit-jupiter-engine</artifactId> <version>5.7.0-RC1</version> </dependency> <dependency> <groupId>org.junit.vintage</groupId> <artifactId>junit-vintage-engine</artifactId> <version>5.7.0-RC1</version> </dependency> </dependencies> <configuration> <statelessTestsetReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5Xml30StatelessReporter"> <usePhrasedFileName>false</usePhrasedFileName> <usePhrasedTestSuiteClassName>true</usePhrasedTestSuiteClassName> <usePhrasedTestCaseClassName>true</usePhrasedTestCaseClassName> <usePhrasedTestCaseMethodName>true</usePhrasedTestCaseMethodName> </statelessTestsetReporter> <consoleOutputReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5ConsoleOutputReporter"> <usePhrasedFileName>false</usePhrasedFileName> </consoleOutputReporter> <statelessTestsetInfoReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5StatelessTestsetInfoReporter"> <usePhrasedFileName>false</usePhrasedFileName> <usePhrasedClassNameInRunning>true</usePhrasedClassNameInRunning> <usePhrasedClassNameInTestCaseSummary>true</usePhrasedClassNameInTestCaseSummary> </statelessTestsetInfoReporter> </configuration> </plugin>

Error trace:
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.203 s - in null
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] com..........TestCase.testGenericClientsFactory
[INFO] Run 1: PASS
[INFO] Run 2: PASS
[INFO] Run 3: PASS
[INFO] Run 4: PASS
[INFO] Run 5: PASS
[INFO] Run 6: PASS
[INFO] Run 7: PASS
[INFO] Run 8: PASS
[INFO] Run 9: PASS
[INFO] Run 10: PASS
[INFO] Run 11: PASS
[INFO] Run 12: PASS
[INFO] Run 13: PASS
[INFO] Run 14: PASS
[INFO] Run 15: PASS
[ERROR] Run 16: TestCase.lambda$null$3:81->testClientExecute:133->lambda$testClientExecute$7:162 » JSON
[INFO] Run 17: PASS
[INFO] Run 18: PASS
[INFO]
[INFO]
[ERROR] Tests run: 33, Failures: 0, Errors: 1, Skipped: 0

Please help me out on this scenario

[Tibor17]

@arun-mano
The textual description has to be enabled in your case. Please read the doc http://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html#Surefire_Extensions_and_Reports_Configuration_for_.40DisplayName

[arun-mano]

@arun-mano
The textual description has to be enabled in your case. Please read the doc http://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html#Surefire_Extensions_and_Reports_Configuration_for_.40DisplayName

Hi Tibor,
I have used the same configuration that you have quoted in- https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html,

Without that config, no logs printed either in console or XML. but after adding your config I got enough details in XML under sure-fire reports folder but in maven build logs alone no displayname based info printed and listed with Index during build errors.

Note: I am using Dynamic Test for Junit

[Tibor17]

Dynamic tests are not fully supported. T Dňa po 14. 9. 2020, 14:26 arun-mano <notifications@github.com> napísal(a):

…

@arun-mano <https://github.com/arun-mano> The textual description has to be enabled in your case. Please read the doc http://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html#Surefire_Extensions_and_Reports_Configuration_for_.40DisplayName Hi Tibor, I have used the same configuration that you have quoted in- https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html , Without that config, no logs printed either in console or XML. but after adding your config I got enough details in XML under sure-fire reports folder but in maven build logs alone no displayname based info printed and listed with Index during build errors. Note: I am using Dynamic Test for Junit — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#990 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH7ER6JHCSS4S7STEFNTXDSFYDXVANCNFSM4DUUSLIA> .

[sblumenstock-doubleSlash]

So Dynamic Tests are not supported with display names in sure-fire plugin?

[oxygenecore]

I have the same issue. Moving this back to maven, they don't particularly care about console logging format, and there is no way to communicate this to them (comments only for registered users, public registrations are closed, I'm not really sure how the bug reporting should work in such environment).
This thing lingers for a while, and it really looks like a bug in JUnit reporting :/

[sbrannen]

@oxygenecore, as far as I can tell, Surefire actually has support for console logging which supports JUnit 5 display names.

Have you looked into that?

https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html#surefire-extensions-and-reports-configuration-for-displayname

[sbrannen]

there is no way to communicate this to them (comments only for registered users, public registrations are closed, I'm not really sure how the bug reporting should work in such environment).

@oxygenecore, you can use the mailing list, and if you still want a JIRA account, you can request one.