Cypress <> Playwright Comparison

[sharmilajesupaul]

So these are just quick thoughts on the benefits we get from Cypress, and where Playwright might be able to bridge the gap. Some of these might be things better supported on Playwright itself and not the runner (like proxy config).

These are things that Cypress has that I'm not sure Playwright does yet (correct me if I'm wrong):

1. A headed in-browser test runner UI - developers can watch and debug tests in real time, this is also hooked up to a "watch" mode that picks up code changes. This may be the largest benefit we receive from Cypress in terms of developer experience. The UI test runner allows you to interact with the DOM and pause the test on any steps. (video eg). This UI also surfaces the test failure messages in recorded videos.

2. Config options to run tests against any base URL -- we point Cypress at any host by changing an env variable CYPRESS_BASE_URL.

3. On the same note as the previous ^, ability to propagate any environment variables into browser tests. Cypress uses a convention where any env var prefixed with CYPRESS_ can be discovered and used by tests. https://docs.cypress.io/guides/guides/environment-variables.html#Setting

4. Ability to listen mock and stub routes and API responses https://docs.cypress.io/api/commands/route.html#Syntax

5. Option to output recorded video of the test runs -- seems like you heard this one already Record a video for tests #5

6. Proxy configuration for in browser tests https://docs.cypress.io/guides/references/proxy-configuration.html#Set-a-proxy-on-Linux-or-macOS

There's probably more that I'm forgetting, but those are the big ones I can think of.

[sharmilajesupaul]

Re proxy support - I just found microsoft/playwright#810, looks like this is on your radar already.

[JoelEinbinder]

Thanks for the great list!

1. A headed in-browser test runner UI - developers can watch and debug tests in real time, this is also hooked up to a "watch" mode that picks up code changes. This may be the largest benefit we receive from Cypress in terms of developer experience. The UI test runner allows you to interact with the DOM and pause the test on any steps. (video eg). This UI also surfaces the test failure messages in recorded videos.

We definitely should have a better debugging story. Filed as #14.

I'm not so sure about combining a debug mode and a watch mode. We could easily just have your tests run in headful, and browsers will pop up and go away with your tests. But doing a good visual test dashboard is probably out of scope of this project, or at least very far away.

2. Config options to run tests against any base URL -- we point Cypress at any host by changing an env variable CYPRESS_BASE_URL.

What does this mean? It changes the url the page navigates to at the start? I think you could emulate that today with like page.goto(process.env.BASE_URL || 'https://www.airbnb.com'). Or is there something more magic to it?

3. On the same note as the previous ^, ability to propagate any environment variables into browser tests. Cypress uses a convention where any env var prefixed with CYPRESS_ can be discovered and used by tests. https://docs.cypress.io/guides/guides/environment-variables.html#Setting

Our e2e tests run in node, and so you can use process.env in them. The unit tests run in the browser, but I can create a fake process.env if its useful. I assume this is mostly for things like API keys?

4. Ability to listen mock and stub routes and API responses https://docs.cypress.io/api/commands/route.html#Syntax

In e2e tests this works with page.route. For unit tests, this is blocked on #13.

[sharmilajesupaul]

What does this mean? It changes the url the page navigates to at the start? I think you could emulate that today with like page.goto(process.env.BASE_URL || 'https://www.airbnb.com'). Or is there something more magic to it?

Using process.env will work! The Cypress framework special cases the base url that the tests will hit, it needs to to be a a global configuration option: https://docs.cypress.io/guides/guides/environment-variables.html#Overriding-Configuration

Our e2e tests run in node, and so you can use process.env in them. The unit tests run in the browser, but I can create a fake process.env if its useful. I assume this is mostly for things like API keys?

Why are there two test modes with this distinction? Are there some parts of the API only available in one test mode or the other?

But yeah, I think it should be fine. We're using the env vars in Cypress tests to do things like, bootstrap pages with some required cookies, adjust the timeout on certain commands in different environments, etc.