test: do graceful shutdown by default

[koivunej]

It should give us all possible allowed_errors more consistently.

While getting the workflows to pass on #8632 it was noticed that allowed_errors are rarely hit (1/4). This made me realize that we always do an immediate stop by default. Doing a graceful shutdown would had made the draining more apparent and likely we would not have needed the #8632 hotfix.

Downside of doing this is that we will see more timeouts if tests are randomly leaving pause failpoints which fail the shutdown.

The net outcome should however be positive, we could even detect too slow shutdowns caused by a bug or deadlock.

[koivunej]

There should be some failures. (I did not get any of these locally.)

[github-actions]

2116 tests run: 2047 passed, 0 failed, 69 skipped (full report)

---

Code coverage* (full report)

• functions: 32.5% (7167 of 22034 functions)
• lines: 50.5% (57920 of 114712 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
f9e4531 at 2024-08-12T12:29:10.334Z :recycle:}

[skyzh]

Just FYI, changes to the shutdown mode was recent -> #8289

[koivunej]

Just FYI, changes to the shutdown mode was recent -> #8289

That's unrelated, this is mostly about pageserver shutdown, and it has been using SIGQUIT at least 2y -- but I closed the blame page already.

[koivunej]

Excellent, the rest reproduce locally.

[skyzh]

early approval as I'm off next week and I expect this patch only changes error handling / failpoints and won't affect the actual code behavior :)

[koivunej]

• test_sliding_working_set_approximation: debug

With the verbose pausable failpoint, it would appear that the logging is now slowing down the test.

Probably the same for the rest.