Run test suite with executionOrder="random"

[Majkl578]

Triggered by https://twitter.com/phpdudeaz/status/1093176110047584257.

My main concern with this approach is determinism. I know it's the sole point of this idea, but debugging these failures locally (or getting them randomly in PRs) is terrifying.
Therefore it may be a good idea to run tests randomly only when if: NOT type = pr.

I'm not sure whether PHPUnit provides seed used for the execution that could later be used for reproducible failure. If not, adding such feature into PHPUnit may be a pre-requisite.

Edit: PHPUnit does provide a way to set the seed, as well as printing it as part of output, but only in verbose mode (sebastianbergmann/phpunit#3515).

[morozov]

In general, I agree that the random order should be only used on release branches, not in PRs. A contributor of a given change doesn't have to solve unrelated issues caused by hidden dependencies in existing tests.

On the other hand, a new PR should be prevented from introducing those dependencies. Therefore, once the tests get stable enough on feature branches, we can try enabling random order on PRs as well.

[morozov]

I took an initial stab at it in #4731. Since there doesn't seem to be a way to make failures optional on GitHub, running tests in random order on CI doesn't seem feasible.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.