Skip to content

Commit

Permalink
docs(contributing): add info about skip and fail (#1944)
Browse files Browse the repository at this point in the history
When contributing some test cases, I was a bit confused about if I should use `skip` and `fail`, the difference between the two, and how they interact with CI/CD tests passing or failing: #1939 (comment).

This is my attempt at explaining the difference, but feel free to edit to make it more clear and concise (and correct any inaccuracies).
  • Loading branch information
rwoll authored Apr 23, 2020
1 parent 05f0797 commit fd17cfb
Showing 1 changed file with 20 additions and 0 deletions.
20 changes: 20 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,26 @@ HEADLESS=false SLOW_MO=500 npm run wtest
BROWSER=chromium node --inspect-brk test/test.js
```

- When should a test be marked with `skip` or `fail`?

- **`skip(condition)`**: This test *should ***never*** work* for `condition`
where `condition` is usually a certain browser like `FFOX` (for Firefox),
`WEBKIT` (for WebKit), and `CHROMIUM` (for Chromium).

For example, the [alt-click downloads test](https://github.com/microsoft/playwright/blob/471ccc72d3f0847caa36f629b394a028c7750d93/test/download.spec.js#L86) is marked
with `skip(FFOX)` since an alt-click in Firefox will not produce a download
even if a person was driving the browser.


- **`fail(condition)`**: This test *should ***eventually*** work* for `condition`
where `condition` is usually a certain browser like `FFOX` (for Firefox),
`WEBKIT` (for WebKit), and `CHROMIUM` (for Chromium).

For example, the [alt-click downloads test](https://github.com/microsoft/playwright/blob/471ccc72d3f0847caa36f629b394a028c7750d93/test/download.spec.js#L86) is marked
with `fail(CHROMIUM || WEBKIT)` since Playwright performing these actions
currently diverges from what a user would experience driving a Chromium or
WebKit.

## Public API Coverage

Every public API method or event should be called at least once in tests. To ensure this, there's a `coverage` command which tracks calls to public API and reports back if some methods/events were not called.
Expand Down

0 comments on commit fd17cfb

Please sign in to comment.