Consider adding a -n (no execute) option

[sharimo1]

TestNG Version

Note: only the latest version is supported

Expected behavior

I guess this is a feature request rather than a bug, but I saw in the testng-users questions that other people have this issue too. Please consider adding a no-execute option - one that will list the tests that will execute in the order they would execute without actually executing them.
This could be useful in cases such as:

• When you have used "dependsOnMethods" to sequence tests
• When you have used include/exclude in the testng.xml file
• when you want see what tests have been tagged with a particular group name
• To see the effects of combinations of the above!

Actual behavior

Is the issue reproductible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

[juherr]

Thanks for the idea.

According to your idea, is the no-execute option supposed to execute (or not) factories and data providers?
Execution of both may have side effects in the system under test but we need them in order to find all test methods.

[tjni]

In systems I've seen, this behavior is usually called a dry run, so, if implemented, I'd like to see the option name be closer to that.

My two cents: If all I'm told is the name of this feature, I would expect it to run everything except for the logic in the tests themselves. That is, tests would be the only side effects that are ignored.

[sharimo1]

This is not an original idea. "make" which is the mother of all build tools had such an option:
https://www.gnu.org/software/make/manual/html_node/Options-Summary.html
Side effects may be necessary.
It would be nice to if a verbose option let me see what triggers a particular test to be included/excluded.

[cbeust]

I'm actually surprised it's not already there, I thought I had implemented that, but maybe I got myself confused with Kobalt, which allows that.

Since both Kobalt and TestNG use the same engine (DynamicGraph), this is actually pretty easy to implement: when comes the time to add the tasks to the graph, just return a Result object directly instead of calling the test method: example.

[krmahadevan]

Inputs from @cbeust

FYI, it would be pretty easy to implement -dryRun:

• Create a subclass of IWorker which, instead of actually running a test, immediately returns with a success status
• Return these workers from the IWorkerFactory.

[cbeust]

FYI, I used this exact same idea in Kobalt, which also uses DynamicGraph: link.

[krmahadevan]

@sharimo1 - This is now available in TestNG 6.14-SNAPSHOT.

To run tests in dry mode, please use the JVM argument -Dtestng.mode.dryrun=true. When run in the dry run mode, TestNG will skip executing any of the @Test methods and also any of the @BeforeXXX/@AfterXXX configuration methods and instead fake them to have passed. But your data provider implementations, listener implementations etc., will still be executed.