À tout de suite: fast self-test suite

[epdenouden]

As a developer of PHPUnit
I want a fast integration test of all basic use cases
so that experimenting with the new logging is easier and more fun

Inspiration

Related to the redesign of the result event and logging system and the updates for test execution order logging and TestDox.

Working on the test execution ordering I wished I had a quick test suite collection of tests that shows me the general state of PHPUnit with a focus on the developer experience. The current test collection is too unwieldy to run on my old laptop continuously.

What functionality to test?

• are all test result status and messages presented to the user; example: JUnit logger skips PHPT tests #3436
• do exceptions and errors bubble to the surface; example: PhpStorm/Teamcity + PhpUnit 7.4.1 - Error in setUpBeforeClass #3364
• running order of test collection within current architecture; Breakfast: rerun defects first #3147 and all its related issues like Unable to chain a @dataProvider in method a with a @depends in method b #3093, Combined use of @depends and @dataProvider is not handled correctly #3156
• buffered logging of async test events to decouple execution order and result printer; example: "Buffered" TestDox printer(s) #3380
• output format of the loggers included with the basic package

Implementation

• create a small test suite with the basics as a part of the main test collection, but which can also be reused for individual end-to-end tests
• reuse existing tests where possible
• review existing end-to-end tests: remove duplicates, obsolete regression tests
• make it easier to test loggers via unit tests

List of scenarios

Basics

• all test result types with standard messages
• all test result types with verbose messages

Fixtures

• TestCase class with a failing setUpBeforeClass()
• TestCase class with a failing tearDownAfterClass()
• TestCase class with a setUp() that fails always
• TestCase class with a tearDown() that fails

Dataproviders

• Test with @dataprovider and a provider that returns a simple array
• Test with @dataprovider and a provider that returns named rows
• Test with an incompatible @dataprovider
• Test with a @dataprovider that returns no data
• Test with a Generator @dataprovider
• Test with a Generator @dataprovider with a duplicate key
• Test with a malformed @dataprovider row
• skipped dataprovider; see DataProviderSkippedTest

Test execution order

• Test that @depends on a previous test (original: no output)
• Test that @depends on a 'later' test (original: skipped, when no-depends)
• Test that @depends on a Test in another class
• unbuffered logging: reordering affects both run order and output order
• buffered logging: reordering affects only run order and result, but not output order
• no difference between buffered and unbuffered logging when in original order
• output random configuration to make random runs are reproducible, see Random order seed should be printed in non-verbose mode as well #3515

Test collection structure

• TestSuite with only nested TestSuites
• TestSuite with only TestCases
• TestSuite with a mix of suites and tests
• Test class without any tests
• PHPT end-to-end
• skipped PHPT end-to-end via a triggered --SKIPIF-- section
• Test skipped by @requires

Isolation

• Test class with @runClassInSeparateProcess, all events reach user
• Test with @runInSeparateProcess, all events reach user

Included printers and logging formats

• --debug
• the classic dot-o-matic printer
• TestDox (with all features)
• TeamCity
• JUnit
• JUnit v5 see Support new standard for reporting test results in XML format #3162
• the formats above with and without colorization
• the formats above with both quiet and verbose

[epdenouden]

@sebastianbergmann forgot to add, I am already working on this. You can assign the issue as a xmas present.