Add #define to not run tests by default

[tbleher]

Description

Currently, tests are run by default, but can be disabled via --dt-no-run. This is of course preferred for test suites, but is not so good if one wants to ship tests with the production code.
I propose adding a #define to invert this behavior. Something like DOCTEST_CONFIG_NO_RUN_DEFAULT.
Adding this define would have the following effect:

• Instead of a --dt-no-run option, there would be a --dt-run option.
• When the --dt-run option is not provided, no_run is set, and the test suite is not run.
• When the --dt-run option is provided, no_run is set to false, and the test suite is run as normal.

I believe this would be beneficial for including tests in production code.

[tbleher]

Note: I'm willing to provide code to support this feature, but first wanted to check if something like this would be accepted (according to the contributor guidelines).

[onqtam]

Checkout how to define your own main function - you can change the default value for this setting and still have the ability to override the setting through the command line and run the tests.

You could even make your own preprocessor identifier that when defined changes the default value for this option - so only “release” or “production” builds have the tests disabled by default (not ran by default but still compiled - for not even compiling them checkout DOCTEST_CONFIG_DISABLE)

I feel that this is best left to the users since if you have tests and production code alongside each other you are definitely providing the main entry point of the program and further complicating the framework with more config options is unnecessary.

It is nice to hear that people are actually using doctest in this way!

I’m closing this but if you think there is more to it - feel free to reopen it.

[tbleher]

Thanks for the pointer on how to change the default value in the main function! I'm fairly new to doctest, and I hadn't understood that correctly. So I agree that the default value itself doesn't need a #define. However, this still leaves the command line options a bit strange. In the scenario I mentioned, people would enable the tests via

$ ./executable --dt-no-run=false

which seems quite unnatural. One could of course have an additional command line parser outside of doctest, which acted on something like --dt-run. I currently plan to do it like that. This means, however, that the new command line option is not listed alongside the other doctest options, which makes the user interface a bit inconsistent.
I'm currently working on some other small issues related to command line handling, and then might revisit this topic.