From fd17cfb5c1eb54147c4d90a73350b4a926e3bd21 Mon Sep 17 00:00:00 2001 From: Ross Wollman Date: Thu, 23 Apr 2020 14:28:40 -0700 Subject: [PATCH] docs(contributing): add info about skip and fail (#1944) When contributing some test cases, I was a bit confused about if I should use `skip` and `fail`, the difference between the two, and how they interact with CI/CD tests passing or failing: https://github.com/microsoft/playwright/pull/1939#discussion_r413946012. This is my attempt at explaining the difference, but feel free to edit to make it more clear and concise (and correct any inaccuracies). --- CONTRIBUTING.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e97fd320bae43..615d21ab0c713 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -198,6 +198,26 @@ HEADLESS=false SLOW_MO=500 npm run wtest BROWSER=chromium node --inspect-brk test/test.js ``` +- When should a test be marked with `skip` or `fail`? + + - **`skip(condition)`**: This test *should ***never*** work* for `condition` + where `condition` is usually a certain browser like `FFOX` (for Firefox), + `WEBKIT` (for WebKit), and `CHROMIUM` (for Chromium). + + For example, the [alt-click downloads test](https://github.com/microsoft/playwright/blob/471ccc72d3f0847caa36f629b394a028c7750d93/test/download.spec.js#L86) is marked + with `skip(FFOX)` since an alt-click in Firefox will not produce a download + even if a person was driving the browser. + + + - **`fail(condition)`**: This test *should ***eventually*** work* for `condition` + where `condition` is usually a certain browser like `FFOX` (for Firefox), + `WEBKIT` (for WebKit), and `CHROMIUM` (for Chromium). + + For example, the [alt-click downloads test](https://github.com/microsoft/playwright/blob/471ccc72d3f0847caa36f629b394a028c7750d93/test/download.spec.js#L86) is marked + with `fail(CHROMIUM || WEBKIT)` since Playwright performing these actions + currently diverges from what a user would experience driving a Chromium or + WebKit. + ## Public API Coverage Every public API method or event should be called at least once in tests. To ensure this, there's a `coverage` command which tracks calls to public API and reports back if some methods/events were not called.