Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PHPUnit 10 can we get (back?) the summary of non-successful tests when using testdox #5318

Closed
Ilyes512 opened this issue Apr 11, 2023 · 28 comments
Labels
type/enhancement A new idea that should be implemented

Comments

@Ilyes512
Copy link

Ilyes512 commented Apr 11, 2023

When you use PHPUnit 10 with testdox you don't get a summary anymore at the end. When you got 100's of tests you will need to scroll up and search for the failed test that might be scattered.

I have 2 example outputs of running a test (with failures) in Laravel 9 which uses PHPUnit 9 and Laravel 10 which uses PHPUnit 10:

Laravel 9 with PHPUnit 9:
phpunit-9

Laravel 10 with PHPUnit 10:
phpunit-10

So of course whenever you only got 2 tests it doesn't matter, but if you got 100's of tests it's allot harder to get an overview of the failed tests.

This might be related to #5304

@Ilyes512 Ilyes512 added the type/enhancement A new idea that should be implemented label Apr 11, 2023
@sebastianbergmann
Copy link
Owner

I do not think that this summary of non-successful tests is useful.

@Ilyes512
Copy link
Author

Ilyes512 commented Apr 12, 2023

To bad because this is clearly something that made the DX worse (when having a production application with 1000's of tests using testdox in CI). 🤷

@MelchiorKokernoot
Copy link

Agreed. This summary is much needed

@jmckenna
Copy link

Seconded, this summary is still very needed.

@bilogic
Copy link

bilogic commented Apr 29, 2023

I do not think that this summary of non-successful tests is useful.

@sebastianbergmann Could you elaborate more? Is there a better way to quickly find out which test (among the hundreds) has failed?

@robertmarney
Copy link

@sebastianbergmann - For additional context: Some of our test suites have 5000 tests, reading the inline logging in testdox is a major downgrade in experience compared to the previous versions. In the new way we need to review sometimes 15000 lines of results to find the details of the failure :(

@Minhyme
Copy link

Minhyme commented Jul 11, 2023

Quick and dirty for people to add to their CI to somewhat restore functionality:

phpunit --testdox > test ; cat test ; cat test | grep "│\|✘" ; if grep -q "ERRORS\|FAILURES" test; then rm test; exit 1; else rm -f test; fi

@lk77
Copy link

lk77 commented Aug 2, 2023

that summary was really usefull, good luck finding the failing tests now, i have more than 1000 tests

@sfsaccone
Copy link

You can grep failed tests from --testdox-text output (grep '\[ \]' <file>), but that only shows test names, not the files. Is there a log format that puts everything on one line? Otherwise, it must be scripted.

I, too, miss the summary.

I may go back to using PHPUnit Text Explorer in VS Code. It includes an option to show only the failed tests. Running tests via the command line seemed easier than the explorer when --testdox included the summary, but not anymore.

@eric-pfi
Copy link

+1 for wanting the summary. When Jenkins runs fail, my default is to look at the end of the console output to see the summary of failed tests; having to search for them is extra unnecessary work.

@lk77
Copy link

lk77 commented Aug 24, 2023

I have a tip to help find the failing tests :

ctrl+f and search for the ✘ symbol, it's unicode U+2718

on linux, hold left ctrl + shift + u until you have a underlined u and type 2718 then enter, and you should have the ✘ character

@borys-p
Copy link

borys-p commented Aug 29, 2023

@sebastianbergmann is there a good reason to remove such a useful behavior? If people need to search for unicode characters in the output, then something went terribly wrong.

@tarlepp
Copy link

tarlepp commented Oct 12, 2023

Yeah, that old summary was really useful because you could see those errors easily from that summary - now we need to scroll up to see which test failed and this is really pita when you've lot of test like in my case 5k+

@lk77
Copy link

lk77 commented Oct 17, 2023

We had no other choice but to revert to phpunit 9

@tarlepp
Copy link

tarlepp commented Dec 30, 2023

We had no other choice but to revert to phpunit 9

Using phpunit 9 is not a real solution - sooner or later you need to upgrade.

The proper solution - imho - would be one of following:

  • Bring this feature back to phpunit itself
  • Community plugin/extension that add this feature

And - imho - the best solution would be bringing this back to phpunit itself.

I don't recall to see any deprecation notices in this summary within phpunit 9 - @sebastianbergmann should you first deprecate features like this before just removing those?

@sebastianbergmann
Copy link
Owner

As discussed in #5488, #5518, and #5521: this will be brought back. In fact, this is the first thing I plan to work on in the new year after a couple of days off.

@sebastianbergmann
Copy link
Owner

sebastianbergmann commented Dec 30, 2023

Sorry for the noise, my bad. This is not about the TestDox issues tracked in #5488, #5518, and #5521 like I initially understood. What I wrote back in #5318 (comment) still holds: I do not think that summary of non-successful tests is useful.

However, I will consider a PR that brings this summary of non-successful tests back as an opt-in.

@tarlepp
Copy link

tarlepp commented Dec 30, 2023

I do not think that summary of non-successful tests is useful.

So when we have eg. that 5k+ tests and some of those fails, we really need to scroll up all those testdox output to find out which of those tests really failed - it's really bad ux - imho.

@bilogic
Copy link

bilogic commented Dec 30, 2023

@tarlepp

So when we have eg. that 5k+ tests and some of those fails, we really need to scroll up all those testdox output to find out which of those tests really failed - it's really bad ux - imho.

This example has been raised multiple times, but the answer remains the same without really addressing why.

However, I will consider a PR that brings this summary of non-successful tests back as an opt-in.

I hope you read this line too, because there is a ray of hope now.

@tarlepp
Copy link

tarlepp commented Dec 31, 2023

I hope you read this line too, because there is a ray of hope now.

Yeah, I read that. The point of my comment was just to point out that bad ux with current implementation.

And sure it would be nice to see this feature back (by default or opt-in), but we don't have that yet and as you said there might be some hope for that now.

This example has been raised multiple times, but the answer remains the same without really addressing why.

Yep it would be really nice to have more detailed answer why "it's not useful" for developer point of view that is the most important thing to see - imho

@epdenouden
Copy link
Contributor

epdenouden commented Dec 31, 2023

@sebastianbergmann wrote

However, I will consider a PR that brings this summary of non-successful tests back as an opt-in.

I will build a proposal for the Testdox summary. Nice way to dive into all the refactoring that has been done.

When working on the refactoring of the colorization internals I run into workflows where I had one unit test fail as expected and then a small zoo of end-to-end tests that use Testdox for validation testing. Happy scrolling:

[test output for 3407 plus suite headers]

FAILURES!
Tests: 3407, Assertions: 8991, Failures: 13, Skipped: 16.

@kocsismate
Copy link

kocsismate commented May 14, 2024

Just to give one more example where a summary would be very useful: we have ~13k tests, and it's extremely tedious to find failures where there is only a few of them. We use GitLab, and it truncates PHPUnit output, so we have to rely on the raw output, but that's also difficult parse when colors are enabled.

Originally, we enabled the testdox based output with PHPUnit 9 in order to be able to track fatal errors more easily when the execution of PHPunit is halted immediately, but as far as I could determine it's also a regression in PHPUnit 10 that the testdox output is delayed at the end of the execution so we cannot see which test resulted in the fatal error.

@sebastianbergmann
Copy link
Owner

we enabled the testdox based output with PHPUnit 9 in order to be able to track fatal errors more easily when the execution of PHPunit is halted immediately

The --debug CLI option is far better suited than TestDox output for tracking fatal errors.

@pooriaazimi
Copy link

pooriaazimi commented Jun 6, 2024

I agree with the other comments. The summary of errors at the end is by far the most important part of PHPUnit output! We have tens of thousands of unit tests that take a long time to execute. They are run in GitHub Actions after every commit, and we often check back in 5-10 minutes to see which tests failed. In PHPUnit 9, clicking on the "Test Failed" action in GitHub would helpfully take us to the end of the logs with a list of all failing tests. Now we have to spend minutes scrolling through the full list, searching for the ✘ unicode character...

@juergenZeindlinger
Copy link

@sebastianbergmann why you don't bring back that summary? As you can see in this thread the people like it and need it. I agree with other comment's that it's really not a good thing to simply remove but not mark functionality as deprecated.

@MazzMazz
Copy link

MazzMazz commented Jul 1, 2024

@sebastianbergmann The summary is also very much needed on our side and i can completely agree with @kocsismate . The developer experience decreased a lot after upgrading to PHPUnit10

@sebastianbergmann
Copy link
Owner

I am sorry about the confusion and frustration I caused with my comments in #5318 (comment) and #5318 (comment).

I never imagined that the TestDox output would be "permanently" configured/enabled in a project. To me, that does make sense, especially in a CI environment. To me, TestDox is (and always was) about documentation, and not about "a different way of showing test results".

I now realize that that many developers actually want to always use the TestDox output and therefore now have a worse user experience than before due to the fact that the TestDox output for non-successful tests is not repeated at the end.

The previous functionality will be brought back as an option (configurable through configuration file and CLI option). The work on this is tracked in #5885.

@bilogic
Copy link

bilogic commented Jul 1, 2024

Thanks, among all the various PHPUnit outputs, testdox is the most readable (what happened and what went wrong), which I guess is why so many are asking for its return.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement A new idea that should be implemented
Projects
None yet
Development

No branches or pull requests