Playwright: run tests headlessly.

[worldomonation]

Changes proposed in this Pull Request

This PR proposes to run Playwright tests headlessly.

By some accounts, Playwright runs faster in headless mode GitHub.

Anecdotally, running a selection of test specs locally with both headful and headless modes show some difference:

Headless:

  [WPCOM] Blocks: CoBlocks: (mobile) @parallel
    ✓ Log in (2541 ms)
    ✓ Start new post (5238 ms)
    ✓ Enter post title (274 ms)
    ✓ Insert Pricing Table block and enter price to left table (2098 ms)
    ✓ Insert Dynamic HR block (996 ms)
    ✓ Insert Hero block and enter heading (1227 ms)
    ✓ Insert Click to Tweet block and enter tweet content (1163 ms)
    ✓ Insert Logos block and set image (2063 ms)
    ✓ Publish and visit post (7967 ms)
    ✓ Confirm Pricing Table block is visible in published post (35 ms)
    ✓ Confirm Dynamic HR block is visible in published post (24 ms)
    ✓ Confirm Hero block is visible in published post (76 ms)
    ✓ Confirm Click to Tweet block is visible in published post (57 ms)
    ✓ Confirm Logos block is visible in published post (29 ms)

Test Suites: 1 passed, 1 total
Tests:       14 passed, 14 total
Snapshots:   0 total
Time:        27.382 s, estimated 32 s

Headful:

  [WPCOM] Blocks: CoBlocks: (mobile) @parallel
    ✓ Log in (2871 ms)
    ✓ Start new post (5304 ms)
    ✓ Enter post title (503 ms)
    ✓ Insert Pricing Table block and enter price to left table (1691 ms)
    ✓ Insert Dynamic HR block (773 ms)
    ✓ Insert Hero block and enter heading (1126 ms)
    ✓ Insert Click to Tweet block and enter tweet content (1072 ms)
    ✓ Insert Logos block and set image (2098 ms)
    ✓ Publish and visit post (7711 ms)
    ✓ Confirm Pricing Table block is visible in published post (44 ms)
    ✓ Confirm Dynamic HR block is visible in published post (24 ms)
    ✓ Confirm Hero block is visible in published post (31 ms)
    ✓ Confirm Click to Tweet block is visible in published post (18 ms)
    ✓ Confirm Logos block is visible in published post (26 ms)

Test Suites: 1 passed, 1 total
Tests:       14 passed, 14 total
Snapshots:   0 total
Time:        31.934 s, estimated 42 s

Testing instructions

• desktop tests pass
• mobile tests pass
• pre-release tests pass

[github-actions]

Link to Calypso live: https://calypso.live?image=registry.a8c.com/calypso/app:build-15458
Link to Jetpack Cloud live: https://calypso.live?image=registry.a8c.com/calypso/app:build-15458&env=jetpack

[matticbot]

This PR does not affect the size of JS and CSS bundles shipped to the user's browser.

Generated by performance advisor bot at iscalypsofastyet.com.

[worldomonation]

To be honest, I am not sure if the headless tests are any faster in CI.

TeamCity makes it difficult to run multiple runs of the same task, so I've manually triggered a few instances to evaluate execution speed:

desktop

run	time
4724	202
4727	133
4729	141

mobile

run	time
3833	173
3836	154
3837	143

It appears that this is faster than the headful variant (compared to trunk), though trunk/headful/desktop on a good day can achieve speeds of ~150s. This is compared to average of ~180s (quarterly average excluding failures for September).

On mobile, it is a similar story where trunk/headful/mobile can achieve speeds of up to ~140s on a good day, but the average tends to be ~168s (quarterly average excluding failures for September).

[worldomonation]

Given the (admittedly small sample size of) data above, I think it is worth trying to run our tests headlessly for a gain of approx. 17%.

[scinos]

Interesting! Does it still capture screenshots and video recordings?

Doc for Chrome suggests we can drop Xvfb (https://developers.google.com/web/updates/2017/04/headless-chrome#faq)

[worldomonation]

Interesting! Does it still capture screenshots and video recordings?

It should - it does locally, but CI may behave differently. I will introduce some failures intentionally to see if it behaves as expected.

Doc for Chrome suggests we can drop Xvfb (https://developers.google.com/web/updates/2017/04/headless-chrome#faq)

Great, I will also take a look in subsequent commits.

[dpasque]

Wow, that's a pretty huge gain! This is awesome, thanks for looking into this @worldomonation! 😄

[worldomonation]

xvfb can be dropped and artifacts are saved!

[cometgrrl]

Looks good to me!

Excited to see things are getting faster with this PR!

[worldomonation]

Looks like I broke the e2e tests with this merge. Not sure why that is, especially since all e2e tests passed in the branch.

Revert is prepped at #56679.