Consider Playwright for E2E testing

[eliot-akira]

Regarding:

• E2E tests with cypress #802

Recently I read some articles on this topic, and learned that both WordPress core and Gutenberg team are moving to Playwright for their E2E tests. I don't have much experience or opinion on the matter, just learning to set things up for my own projects, but from what I've read, it seems Playwright has some advantages over Cypress.

Here's the announcement post about the migration plan from existing test setup using Puppeteer.

• Migrating WordPress E2E tests to Playwright (March 2022)

  There's a new package of testing utilities under development, called @wordpress/e2e-test-utils-playwright.

Here's the ongoing project to migrate tests in Gutenberg.

• Migrate E2E tests to Playwright

  And an older issue where they explain the rationale for choosing Puppeteer over Cypress.

  I've wanted to like Cypress, and it does have a nice interface, but writing tests has been a constant source of headaches for me. I've attributed this to a few main causes..

  • Testing: Experiment with Puppeteer for E2E testing gutenberg#5618

---

A couple of articles comparing the two.

Cypress vs Playwright: A Detailed Comparison

Migrating from Cypress to Playwright

Some of the claimed benefits of Playwright: better performance, developer experience; intercepting native input events, file up/download, network activity; support for multiple browser engines, mobile device emulation.

---

I think the fact that WordPress core and Gutenberg team are using Playwright is in itself an argument for choosing it over Cypress, so that contributors to those projects don't have to learn another testing framework to be able to contribute to the Playground project.

Since the E2E tests for Playground have just been added, it seems like a good opportunity to consider an alternative, before there's a whole suite of tests.

[adamziel]

Thank you for bringing this up @eliot-akira! I was very torn about Playwright vs Cypress so I tried building these E2E tests using both libraries to see what's the difference in practice. Here's my Playwright branch.

Consider this test written in Playwright and then in Cypress:

Playwright

test('Switches WordPress version to ' + version, async ({ page }) => {
	await page.goto('/');
	await setWordPressUrl(page, '/wp-admin');
	// Update settings in Playground configurator
	const configurator = await page.waitForSelector('button#configurator');
	configurator.click();

	const wpVersionSelect = await page.waitForSelector('select#wp-version');
	await wpVersionSelect.selectOption(version);
	await page.click('#modal-content button[type=submit]');
	// Wait for the page to finish loading
	await page.waitForURL(new RegExp(`&wp=${version}`));

	// Go to wp-admin
	const wpFrame = await wordPressIFrame(page);
	if (version === 'nightly') {
		const footer = await wpFrame.waitForSelector('#footer-upgrade');
		expect(await footer.textContent()).toContain(
			'You are using a development version'
		);
	} else {
		await wpFrame.waitForSelector(
			'body.branch-' + version.replace('.', '-')
		);
	}
});

Here's what I like about Playwright:

• It has great tools for replaying and debugging tests
• There's a lot of prior art in WordPress and contributors are familiar with Playwright
• @wordpress/scripts has tools for running Playwright tests

Here's what I didn't like about the Playwright version of those tests:

• It's much more verbose than the Cypress version below. You need to await everything and split what appears to be a "single thought" into multiple lines of code because that's what the syntax requires.
• They took me much longer to write than the cypress ones. The await-ing isn't just a visual burden, but it makes me think a lot more about the order in which things are happening. Intuitively I'd say that's a good thing, but in practice I found it to be a speedbump.
• I couldn't get it to reliably work with the nested iframes structure. Playground.wordpress.net embeds the /remote.html file in an iframe, and inside that there is another iframe where WordPress is rendered. I tried for a few hours, but still couldn't get Playwright to wait for and target the most recent page in the doubly nested WordPress iframe.

Let's also consider some of the Playwright upsides from that comparison article:

Cross-Browser Support

Cypress also seems to support Firefox, Chrome, Edge, and other browsers.

Network Activity Interception

Cypress seems to support that as well.

Mobile Device Emulation

Cypress seems to be limited to the viewport size emulation here. It can perhaps be extended, but it sounds like Playwright is better equipped for that.

File Download and Upload

Playground has a Cypress test for exporting and importing a Playground instance via zip download/upload. It crashes in GitHub CI because of memory constraints, though. I think it can be fixed, but it would be nice not to have to deal with that.

Cypress

let versionMessage = 'Version ' + version;
if (version === 'nightly') {
	versionMessage = 'You are using a development version';
}
it('should switch WordPress version to ' + version, () => {
	// Update settings in Playground configurator
	cy.get('button#configurator').click();
	cy.get(`select#wp-version option[value="${version}"]`).should(
		'exist'
	);
	cy.get('select#wp-version').select(`${version}`);
	cy.get('#modal-content button[type=submit]').click();
	// Wait for the page to finish loading
	cy.url().should('contain', `&wp=${version}`);

	// Go to phpinfo
	cy.setWordPressUrl('/wp-admin');
	cy.wordPressDocument()
		.find('#footer-upgrade')
		.should('contain', versionMessage);
});

Here's what I like about Cypress:

• Every line sounds like a complete thought, almost like a sentence. "Get the configurator button and click it". I can glance at the test and immediately understand the intention.
• Cypress implicitly waits for each element to appear, which allows me to focus on the flow. I was able to just express a sequence of events I'd do as a user without thinking too much about "there's a brief animation here so I need to waitForSelector".
• It has great tools for running and debugging those tests. I'm not sure about replaying after the fact, though, maybe there's a tool for that or maybe that's where Playwright shines.
• The tests are executed in the same JS context as the Playground website which seems to make them more reliable.

Here's what I didn't like about Cypress:

• Higher barrier to entry. There isn't that much prior art in WordPress and the delayed execution model with all the chaining could be confusing to people who aren't familiar with Cypress.

Let's also consider some of the Cypress shortcomings from that comparison article:

Inability to instantiate multiple browsers simultaneously
Lack of support for multi-tab testing.

I don't see a good use-case for doing either of those in the Playground test suite. Perhaps one will emerge in the future, but I don't anticipate that today.

JavaScript is the preferred language for writing test cases using Cypress.

Cypress seems to have great support for types, the existing test suite is all TypeScript.

Conclusion

Cypress was a clear winner to me. At the same time, we don't have many E2E tests yet, as you noticed. There may still be very good reasons to switch to Playwright and I'm very open to that possibility. At the moment, though, I'm more convinced about Cypress.

[eliot-akira]

Wow, that's a deep dive comparing Playwright and Cypress, considering them from various aspects, pros and cons. That's true, the latter test does read nicely, compact with natural flow of actions.

What stood out from the arguments against Playwright: in addition to taking longer to write the tests (fluency and intuitiveness):

couldn't get it to reliably work with the nested iframes structure.. tried for a few hours..

That sounds like the decisive point, if it doesn't work well for the specific needs of the Playground project. I think that concludes the matter, since there's no use fighting the tool when there's already a working testing setup started.

[adamziel]

Sounds good! I'll close this issue for now, then.

[adamziel]

Actually, @WunderBart shared this version of the Playwright test that looks much more appealing:

test( 'Switches WordPress version to ' + version, async ( { page } ) => {
    await page.goto( '/' );
    await setWordPressUrl( page, '/wp-admin' );
 
    // Update settings in Playground configurator
    await page.locator( 'button#configurator' ).click();
    await page.locator( 'select#wp-version' ).selectOption( version );
    await page.locator( '#modal-content button[type=submit]' ).click();
 
    // Go to wp-admin
    const wpFrame = page.frameLocator( 'iframe' );
    if ( version === 'nightly' ) {
        await expect( wpFrame.locator( '#footer-upgrade' ) ).toContainText(
            'You are using a development version'
        );
    } else {
        await expect(
            wpFrame.locator( 'body.branch-' + version.replace( '.', '-' ) )
        ).toBeVisible();
    }
} );

So maybe let's give Playwright another chance provided it can reliably handle the doubly nested iframes used in Playground!

[WunderBart]

So maybe let's give Playwright another chance provided it can reliably handle the doubly nested iframes used in Playground!

It should be pretty straightforward by using frameLocator! You can pierce any number of iframes with it, like this:

const block = page
    .frameLocator( 'iframe#parent-frame' )
    .frameLocator( 'iframe#child-frame' )
    .frameLocator( 'iframe#grandchild-frame' )
    // etc.
    .locator( '.block' );
 
await block.click();

[adamziel]

Okay, I'm on board. I would love to give Playwright one more try! I don't have any bandwidth to actually explore that – any volunteers? :-)

[adamziel]

Additional reasons to use Playwright:

• It's aligned with WordPress and Gutenberg, there's community mindshare and standard tools
• @WunderBart explores recording and replaying Playground test runs as Playwright E2E tests, it would create instant value for Playground core

[bgrgicak]

I have a PR that adds Playwright tests to Playground.

I'm still playing with the configuration and fixtures, but it's working with the iframe.

[eliot-akira]

A recent article on this topic:

• Why we switched from Cypress to Playwright (Sept 2024)

They describe in depth their reasoning, comparing features, performance, API, browser support, etc.