Retries per specfile #2930
Retries per specfile #2930
Conversation
Previously only test-level and suite-level retries were available, which are fine in most cases. In any test which involves state, such as on a server or in a db, the state may be left invalid once the test fails the first time. Any subsequent retries may have no chance of passing due to the invalid state they would have to start with. A new browser instance is created for each specfile, which makes this an ideal place to hook and setup any other states (server, db). Retries on this level would mean that the whole setup process could simply be repeated as if it were for a new specfile, with the results of the failing specfile (that is to be retried) being discarded.
This allows the reporter to set the start and end properties of runnables to the time the event happened as opposed to the time the reporter got notified of the event.
|
@christian-bromann anything that still needs doing here? |
|
@bennieswart two questions:
|
|
@christian-bromann when there are retries left, messages are buffered in a list and only reported once (and if) the tests for that child process have passed. If the child process ends with a failure, the buffered messages are discarded. If no retries are left, messages are unbuffered and reported as they are received. This way, only one set of messages get reported. Regarding v5, I see no reason why we should not port it to v5 also. There have been quite a few interested parties on the previous PR, so it seems to be a practical feature from which others could benefit as well. |
|
This will be really nice to have. |
|
|
|
Very interesting feature! |
|
Hello This feature is really good because it allows running tests in a new environment. For those who want to get it now, there is an alternative implementation of a test runner which works fine. The one drawback of the runner is that wdio's sync option is not supported there. It's the main reason why I don't want to use that in my projects. Literally it means that I have rewrite hundreads of tests. @bennieswart @christian-bromann |
|
This was closed because I moved everything related to v4 to https://github.com/webdriverio-boneyard/v4 |
This is a continuation of #2556.
Proposed changes
Add retries on a per-specfile basis, as discussed in #2484.
Previously only test-level and suite-level retries were available, which
are fine in most cases. In any test which involves state, such as on a
server or in a db, the state may be left invalid once the test fails the
first time. Any subsequent retries may have no chance of passing due to
the invalid state they would have to start with.
A new browser instance is created for each specfile, which makes this an
ideal place to hook and setup any other states (server, db). Retries on
this level would mean that the whole setup process could simply be
repeated as if it were for a new specfile, with the results of the
failing specfile (that is to be retried) being discarded.
Types of changes
Checklist
Further comments
Reviewers: @christian-bromann