introduce global configuration for retry setting

[lnhrdt]

Clear and concise description of the problem

Ideally, automated tests are perfectly deterministic based on the test and source code. In reality, often times tests can be influenced by external factors, especially when they make use of networks or resource intense technologies (e.g. browsers). This can result in "flakey tests." While flakey tests aren't ideal and there are strategies to avoid them, pragmatically many teams must find ways to work with them.

A common strategy is to implement automatic retry logic. If false negatives are somewhat unlikely, retries can help work around them so failing the test suite is statistically negligible.

Often these scenarios only arise in resource constrained environments, like a test runner in CI. When this is the case a solution can be to configure a global retry for that environment so that if any test fails, it will be retried, but only where the failure can be likely attributed to a false negative. #1929 introduced a retry option, but only at the test level which doesn't help in this scenario as it would be tedious to configure for every test, based on an environment.

Thanks @sheremet-va for the suggestion to open this enhancement issue and continue this conversation.

Suggested solution

Introduce a retry option into the global configuration that can be configured statically like this:

import { defineConfig } from 'vitest/config'

export default defineConfig({
  test: {
    retry: 3,
    // ...
  },
})

or based on environment like this:

import { defineConfig } from 'vitest/config'

export default defineConfig({
  test: {
    retry: parseInt(process.env.CI ? 3 : 0),
    // ...
  },
})

Alternative

No response

Additional context

I've also noticed the test-level retry option introduced in #1929 is undocumented, or at least I could not find it. If this feature proposal is accepted, perhaps adding documentation for both retry configurations can be addressed at that time.

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[sheremet-va]

I think to make it easier to use in CI, we can expose --retry option when running Vitest via CLI.

[lnhrdt]

I think to make it easier to use in CI, we can expose --retry option when running Vitest via CLI.

Yes it would be great to have both CLI and config file options. And that would be consistent with other options we use this way in CI like testTimeout.