Support mocha test retries / retry test failures

[cameronc56]

Is this a Feature or Bug?

Feature

Current behavior:

With this.retries set, when a test fails:

TypeError: Cannot set property 'err' of undefined
    at Reporter.mergeErr (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/reporter.js:95:18)
    at Reporter.parseArgs (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/reporter.js:199:20)
    at Reporter.emit (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/reporter.js:190:23)
    at Object.server.startWebsockets.onMocha (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/project.js:287:22)
    at Socket.<anonymous> (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/socket.js:237:36)
    at emitThree (events.js:116:13)
    at Socket.emit (events.js:194:7)
    at /home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/socket/node_modules/socket.io/lib/socket.js:503:12
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

TypeError: Cannot set property 'err' of undefined
    at Reporter.mergeErr (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/reporter.js:95:18)
    at Reporter.parseArgs (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/reporter.js:199:20)
    at Reporter.emit (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/reporter.js:190:23)
    at Object.server.startWebsockets.onMocha (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/project.js:287:22)
    at Socket.<anonymous> (/home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/server/lib/socket.js:237:36)
    at emitThree (events.js:116:13)
    at Socket.emit (events.js:194:7)
    at /home/cameronc/switch/test/matcher/node_modules/cypress/dist/Cypress/resources/app/packages/socket/node_modules/socket.io/lib/socket.js:503:12
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

Desired behavior:

this.retries is supported at the describe() and it() level

[brian-mann]

We've internally talked about this before and we'll be doing something different from the way mocha implements retries.

It will work the same - but will be controlled by different API's than what mocha provides. I'll open a proposal later and link it to this issue.

[dcastil]

Would love to see that feature, also to enable that globally for all tests, e.g. through cypress.json.

[spateluber]

+1 for this feature, along with this it would also be nice to allow global and override retries in mocha it block.

[khuezy]

@brian-mann Hi, do you have any updates on this?

[hdjong1]

@brian-mann Would also very much like an update about this.

[henrymunro]

+1 for this feature. We've seen some occasional failures in our CI that this would really help with.

[EirikBirkeland]

This would be incredibly beneficial. I have a couple of tests that fail at random, and I have not yet figured out how I can get around this. But, if I can simply specify 5 retries, we can have slower yet useful CI builds ...

Quite obviously, even if we are writing Cypress tests idiomatically and correctly, there's no way to get around a flaky server or a flaky app. For this reason, we definitely need a way to retry individual tests. And I don't mind if it's a global toggle, e.g. to just specify max 5 retries for each test

This is by far the most needed feature for us right now. Screenshotting is low prio in comparison (I can hack that with blink-diff or use Percy if needed).
Without stable CI runs, e2e testing is not going to be too popular with my colleagues.

[brian-mann]

We discussed this today as a team. It may be the next feature @chrisbreiding works on after Firefox lands.

[MaaikeFox]

Great feature!
Can you already give some insight in when it will be available? :)

[pacop]

Also waiting for this. @brian-mann any updates?

[kuceb]

You can launch Cypress using the Module API , and effectively collect failed spec files from the run and tell cypress to run just those files. Here's an example:
https://gist.github.com/Bkucera/4ffd05f67034176a00518df251e19f58

Then, instead of cypress run, you use node cypress-retries.js

[EirikBirkeland]

@bkucera That is a great and simple solution. Will give this a try. Thanks for sharing.

[germyjen]

would love this feature to come out of the box for cypress: a global setting to retry failed tests x number of times to reduce timeout flakiness.

@brian-mann any news on where this is with the team?

[bryantabaird]

@bkucera running v3.0.1, I had to change the

const specs = _(results.runs).filter("stats.failures").map("spec.relative").value()
to
const specs = _(results.runs).filter("stats.failures").map("spec.path").value(),
but it worked great for me.

I'm hoping for a way to rerun individual failed tests within a spec, but that didn't seem possible with the current setup, since Cypress parameters only included spec files. I'd love to hear any workarounds for that if you had any.

EDIT: v3.1.0 does use "spec.relative"

[nanoflat]

@bahmutov @brian-mann I think that with retries by workaround #1313 (comment) with Module API will not work with cypress 3.1.x with the parallel run.
The thing that scope of specs will not be same from original run to retry - because some tests will fail and some will pass. So run will fail with https://docs.cypress.io/guides/references/error-messages.html#Cannot-parallelize-tests-across-environments

[kuceb]

@nanoflat I've updated the cypress-retries script to work with Parallelization.

The script should be able to work with parallel runs and non parallel runs. Just pass your configuration to DEFAULT_CONFIG

Here's how it will look on the Dashboard

[itaykotler-fundbox]

@bkucera, your solution works great and I implemented it in my script.
Small problem, Cypress Dashboard marks the run as Failed while the reties succeded.
Is there a way to mark the run as Passed when retries have been successful?

[Francis-Moreau-adsk]

Cypress didn't prove to remove all the 'flaky' test results for us, be we do enjoy the platform a bit more the Selenium webdriver.

The retry feature would be also a critical feature for our automation. We use similar mechanism for our selenium tests and it did save a lot of useless investigation time for false positives.

Hoping for a solution soon!

[EirikBirkeland]

@Francis-Moreau-adsk Feel your pain, man!

Just my 2 cents:
(1) Are you sure the app being tested is robust? I've had flakiness issues too, and I could never pinpoint any exact issue. The back-end was a little slow when testing manually, but not inconsistent.
(2) Could it be that flakiness can be reduced by changing the way you write tests? E.g. avoiding await/async, or making tweaks according to Cypress Promise best practices.

Anyway, I am not using Cypress right now, but having excellent 'Retry' facilities will be important to me in the future.

[Francis-Moreau-adsk]

@EirikBirkeland, thanks
The app we are trying to test is pretty solid but the way it's running there is one Cypress issue (#2552) that causes a lot headaches to write tests.

For sure we need to rethink the way we write tests, and we will get better as we continue trying Cypress. From my short experience with it, I don't see it as a good solution (for us anyways) for true end to end test scenarios. It appears to be viable solution to isolate a small ui component and test it very lightly. Even with the proper Promises in place, we will have to sometimes rely on other mechanism to synchronise the tests (reloads, gets, fixtures, waits...).

For more complex scenarios, there are a lot of shortcuts and workaround involved that real users will never have to do and I'm afraid some bugs might fall in the cracks if Cypress is the only solution to test the UI. Selenium webdriver will still show far more stable test results for us.

As for retries, they are dangerous to use and it's better to make the tests robust before even thinking of a retry mechanism. But I don't see how to completely escape them in a continuous testing build environment.

Still think Cypress is a beautiful beast.

[hilarykitz]

Thirded 🙏

[germyjen]

it's been roughly a year since the original feature request, any update on what's going on with this? Desperate for officially supported retries.

[dwelle]

@germyjen don't think this is a top priority when there's a perfectly working workaround https://github.com/Bkucera/cypress-plugin-retries

[Algram]

Hello there,
I wanted to chime in and try to give you an understanding that this is currently a Number 1 issue and is actually costing you money:

We just had to switch from our 5000$ a year cypress plan to our own parallel implementation. This is due to the issues described above: You can't retry a cypress test reliably in a pipeline when using the parallel feature.

For us this even meant that once you retried a cypress test it always went "green", so we actually deployed failing stuff.

So this issue alone will cose you 5000$ a year now. I would be "SO" happy to pay you for your services, but as it is now, we cannot use your commercial product, even though we want to.

[kuceb]

not trying to minimize the importance of getting this in Cypress core. We hear you, it's up next to be worked on.

But since Github has started collapsing comments in this thread I wanted to make sure people see the workaround plugin posted above:

We're still working on this, but in the meantime I've made a plugin you can try out cypress-plugin-retries:

It supports mocha's retries, and it also retries failures in beforeEach

🚨 Do not comment on this issue about issues with the plugin 🚨

Let me know any bugs in the issues of the plugin repo.

[Algram]

@bkucera this is not really a workaround, because one can still manually retry a test in e.g. GitLab und force it to pass, even if there are real errors that where introduced with this test.

Furthermore it only retries them automatically. Chances are high that the test failure was introduced by a third party system not being available. The plugin would not do anything for that kind of failure.

[lslee714]

The plugin works well for us. @Algram if your tests are dependent on third party systems, you might re-think them... it's generally best practice to mock out external dependencies that you have no control over.

[tiennguyen-ftu-k52]

I'm facing the same issue with @Algram. I'm trying to run cypress parallel on Bitbucket Pipelines and it's always green when you rerun failed step. I tried cypress-plugin-retries but it didn't help.

[sdwvit]

any news on this?

[jm2242]

It looks like the Cypress team is building out native support for retries, so stay tuned!

[msteward]

Would recommend giving the feature an up vote on the Cypress roadmap: https://portal.productboard.com/cypress-io/1-cypress-dashboard/c/59-automatically-retry-failed-specs

[cypress-bot]

The code for this is done in cypress-io/cypress#3968, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

[jennifer-shehane]

Test retries has been released in 5.0.0.

You can refer to our docs on Test Retries for instructions on how to turn on and configure test retries.

There are several options which we think are important to explore depending on your project including configuring globally, per cypress open and cypress run separately, or per test suite / single test.

We think this will be a valuable tool in helping identify flake and have more plans in the future to improve this feature and surface the test retries as well as analytics on flake into our Dashboard.

If you encounter any issues using Test Retries, please open a new issue with a fully reproducible example - do not comment in this thread as this issue is closed.

[csvan]

Awesome work @jennifer-shehane and team, you're the best <3

[kuceb]

for anyone coming from the retries plugin:

Migrating from cypress-plugin-retries to Cypress 5.0.0:

• remove cypress-plugin-retries from devDependencies and related code in support/plugin files
• To enable retries on single test/suite, remove usage of Cypress.currentTest in favor of test config overrides e.g.:

// on a single test
it('test title', { retries: 2 }, () => {
  ...
})

// or on a suite
describe('suite title', { retries: 2 }, () => {
  ...
}) 

• To enable retries globally, set retries in cypress.json instead of using Cypress.env('RETRIES') e.g.:

{ 
  "retries": { "openMode": 0, "runMode": 2 }
}

• remove usage of this.retries(n) (not supported)

[jennifer-shehane]

The Dashboard now displays the screenshots and errors from each attempt of a test that retried. The test results are also filterable by whether they retried.

Filter by whether test retried

See screenshot and error for each attempt when clicking on test