Rework test error handling

[ehuss]

This reworks how errors are handled when running tests and benchmarks. There were some cases where Cargo was eating the actual error and not displaying it. For example, if a test process fails to launch, it only displayed the could not execute process message, but didn't explain why it failed to execute. This fixes it to ensure that the full error chain is displayed.

This also tries to simplify how the errors are handled, and makes them more uniform across test and bench, and with doctests.

This also changes the --no-fail-fast behavior to report errors as they happen instead of grouped at the end (and prints a summary at the end). This helps to make it clearer when a nonstandard error happens. For example, before:

     Running tests/t1.rs (target/debug/deps/t1-bb449dfa37379ba1)

running 1 test
     Running tests/t2.rs (target/debug/deps/t2-1770ae8367bc97ce)

running 1 test
test bar ... FAILED

failures:

---- bar stdout ----
thread 'bar' panicked at 'y', tests/t2.rs:3:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    bar

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

error: test failed.

Caused by:
  process didn't exit successfully: `/Users/eric/Temp/z12/target/debug/deps/t1-bb449dfa37379ba1` (signal: 11, SIGSEGV: invalid memory reference)
  process didn't exit successfully: `/Users/eric/Temp/z12/target/debug/deps/t2-1770ae8367bc97ce` (exit status: 101)

and the changes to that are:

@@ -1,6 +1,10 @@
      Running tests/t1.rs (target/debug/deps/t1-bb449dfa37379ba1)

 running 1 test
+error: test failed, to rerun pass `--test t1`
+
+Caused by:
+  process didn't exit successfully: `/Users/eric/Temp/z12/target/debug/deps/t1-bb449dfa37379ba1` (signal: 11, SIGSEGV: invalid memory reference)
      Running tests/t2.rs (target/debug/deps/t2-1770ae8367bc97ce)

 running 1 test
@@ -18,8 +22,7 @@

 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

-error: test failed.
-
-Caused by:
-  process didn't exit successfully: `/Users/eric/Temp/z12/target/debug/deps/t1-bb449dfa37379ba1` (signal: 11, SIGSEGV: invalid memory reference)
-  process didn't exit successfully: `/Users/eric/Temp/z12/target/debug/deps/t2-1770ae8367bc97ce` (exit status: 101)
+error: test failed, to rerun pass `--test t2`
+error: 2 targets failed:
+    `--test t1`
+    `--test t2`

In the first example, when it says Running tests/t1.rs, there is no error message displayed until after all the tests finish, and that error message is not associated with the original test. This also includes the "to rerun" hint with --no-fail-fast.

[rust-highfive]

r? @weihanglo

(rust-highfive has picked a reviewer for you, use r? to override)

[weihanglo]

I do love this patch. The code is clearer, and also more docs are coming with this PR, not to mentioned tests themselves are more robust.

Thank you for posting this pull request!

[weihanglo]

Although this changes the output of cargo a bit, tools depend on the old output format should be rare. I will move forward and merge it.

@bors r+

[bors]

📌 Commit 23735d4 has been approved by weihanglo

It is now in the queue for this repository.

[bors]

⌛ Testing commit 23735d4 with merge d705fb3...

[bors]

☀️ Test successful - checks-actions
Approved by: weihanglo
Pushing d705fb3 to master...