-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgrade Puppeteer to the latest version #14986
Conversation
This is all failures I see on my local env: Summary of all failing tests
FAIL specs/links.test.js (33.098s)
● Links › link popover remains visible after a mouse drag event
expect(received).toHaveLength(expected)
Expected length: 1
Received length: 0
Received array: []
476 | // The link popover should still be visible
477 | const popover = await page.$$( '.block-editor-url-popover' );
> 478 | expect( popover ).toHaveLength( 1 );
| ^
479 | } );
480 |
481 | it( 'should contain a label when it should open in a new tab', async () => {
at Object.toHaveLength (specs/links.test.js:478:21)
at tryCatch (../../node_modules/regenerator-runtime/runtime.js:62:40)
at Generator.invoke [as _invoke] (../../node_modules/regenerator-runtime/runtime.js:288:22)
at Generator.prototype.(anonymous function) [as next] (../../node_modules/regenerator-runtime/runtime.js:114:21)
at asyncGeneratorStep (specs/links.test.js:15:103)
at _next (specs/links.test.js:17:194)
FAIL specs/preview.test.js (10.067s)
● Preview › should open a preview window for a new post
Error: failed to find element matching selector ".entry-title"
at ElementHandle.$eval (../../node_modules/puppeteer/lib/JSHandle.js:418:13)
-- ASYNC --
at ElementHandle.<anonymous> (../../node_modules/puppeteer/lib/helper.js:110:27)
at DOMWorld.$eval (../../node_modules/puppeteer/lib/DOMWorld.js:149:21)
-- ASYNC --
at Frame.<anonymous> (../../node_modules/puppeteer/lib/helper.js:110:27)
at Page.$eval (../../node_modules/puppeteer/lib/Page.js:329:29)
at Page.<anonymous> (../../node_modules/puppeteer/lib/helper.js:111:23)
at Object._callee2$ (specs/preview.test.js:301:32)
at tryCatch (../../node_modules/regenerator-runtime/runtime.js:62:40)
at Generator.invoke [as _invoke] (../../node_modules/regenerator-runtime/runtime.js:288:22)
at Generator.prototype.(anonymous function) [as next] (../../node_modules/regenerator-runtime/runtime.js:114:21)
at asyncGeneratorStep (specs/preview.test.js:31:103)
at _next (specs/preview.test.js:33:194)
● Preview › should not revert title during a preview right after a save draft
Error: failed to find element matching selector ".entry-title"
at ElementHandle.$eval (../../node_modules/puppeteer/lib/JSHandle.js:418:13)
-- ASYNC --
at ElementHandle.<anonymous> (../../node_modules/puppeteer/lib/helper.js:110:27)
at DOMWorld.$eval (../../node_modules/puppeteer/lib/DOMWorld.js:149:21)
-- ASYNC --
at Frame.<anonymous> (../../node_modules/puppeteer/lib/helper.js:110:27)
at Page.$eval (../../node_modules/puppeteer/lib/Page.js:329:29)
at Page.<anonymous> (../../node_modules/puppeteer/lib/helper.js:111:23)
at Object._callee3$ (specs/preview.test.js:458:32)
at tryCatch (../../node_modules/regenerator-runtime/runtime.js:62:40)
at Generator.invoke [as _invoke] (../../node_modules/regenerator-runtime/runtime.js:288:22)
at Generator.prototype.(anonymous function) [as next] (../../node_modules/regenerator-runtime/runtime.js:114:21)
at asyncGeneratorStep (specs/preview.test.js:31:103)
at _next (specs/preview.test.js:33:194)
● Preview with Custom Fields enabled › displays edits to the post title and content in the preview
Error: failed to find element matching selector ".entry-title"
at ElementHandle.$eval (../../node_modules/puppeteer/lib/JSHandle.js:418:13)
-- ASYNC --
at ElementHandle.<anonymous> (../../node_modules/puppeteer/lib/helper.js:110:27)
at DOMWorld.$eval (../../node_modules/puppeteer/lib/DOMWorld.js:149:21)
-- ASYNC --
at Frame.<anonymous> (../../node_modules/puppeteer/lib/helper.js:110:27)
at Page.$eval (../../node_modules/puppeteer/lib/Page.js:329:29)
at Page.<anonymous> (../../node_modules/puppeteer/lib/helper.js:111:23)
at Object._callee6$ (specs/preview.test.js:619:32)
at tryCatch (../../node_modules/regenerator-runtime/runtime.js:62:40)
at Generator.invoke [as _invoke] (../../node_modules/regenerator-runtime/runtime.js:288:22)
at Generator.prototype.(anonymous function) [as next] (../../node_modules/regenerator-runtime/runtime.js:114:21)
at asyncGeneratorStep (specs/preview.test.js:31:103)
at _next (specs/preview.test.js:33:194)
FAIL specs/demo.test.js
● new editor state › content should load without making the post dirty
expect(jest.fn()).not.toHaveErrored(expected)
Expected mock function not to be called but it was called with:
["Unhandled error"],["TypeError: Cannot read property 'replace' of undefined
at replace (/Users/gziolo/PhpstormProjects/gutenberg/packages/e2e-tests/specs/demo.test.js:20:50)
at Object.responseObjectTransform (/Users/gziolo/PhpstormProjects/gutenberg/packages/e2e-test-utils/build/mocks/@wordpress/e2e-test-utils/src/mocks/mock-or-transform.js:38:38)
at tryCatch (/Users/gziolo/PhpstormProjects/gutenberg/node_modules/regenerator-runtime/runtime.js:62:40)
at Generator.invoke [as _invoke] (/Users/gziolo/PhpstormProjects/gutenberg/node_modules/regenerator-runtime/runtime.js:288:22)
at Generator.prototype.(anonymous function) [as next] (/Users/gziolo/PhpstormProjects/gutenberg/node_modules/regenerator-runtime/runtime.js:114:21)
at asyncGeneratorStep (/Users/gziolo/PhpstormProjects/gutenberg/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
at _next (/Users/gziolo/PhpstormProjects/gutenberg/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
at process._tickCallback (internal/process/next_tick.js:68:7)"]
at expect (../jest-console/build/@wordpress/jest-console/src/index.js:34:4)
FAIL specs/undo.test.js (5.679s)
● undo › Should undo to expected level intervals
expect(received).not.toBeNull()
Received: null
62 | expect( await getEditedPostContent() ).toBe( '' );
63 | // After undoing every action, there should be no more undo history.
> 64 | expect( await page.$( '.editor-history__undo[aria-disabled="true"]' ) ).not.toBeNull();
| ^
65 | } );
66 | } );
67 |
at Object.toBeNull (specs/undo.test.js:64:79)
at tryCatch (../../node_modules/regenerator-runtime/runtime.js:62:40)
at Generator.invoke [as _invoke] (../../node_modules/regenerator-runtime/runtime.js:288:22)
at Generator.prototype.(anonymous function) [as next] (../../node_modules/regenerator-runtime/runtime.js:114:21)
at asyncGeneratorStep (specs/undo.test.js:9:103)
at _next (specs/undo.test.js:11:194)
Test Suites: 4 failed, 66 passed, 70 total
Tests: 6 failed, 4 skipped, 371 passed, 381 total
Snapshots: 252 passed, 252 total
Time: 555.843s
Ran all test suites. |
Did some debugging on the failure of For some reason, in this version of Chromium, the link popover's container is narrower with its content overflowing. When the test performs the drag event, it is actually clicking outside of the popover because of the size issue, which results in dismissal of the popover: I wonder if this is actually a bug in this version of chromium (75.0.3738.0). I downloaded a later version (75.0.3768.0) and checked manually and the popover is normally sized. (Conjecture) It could be that the test failures are related to the version of chromium more so than the puppeteer api. I noticed that the old version of puppeteer we were using was downloading chromium revision r575458, while the upgraded version downloads r641577. |
I just tried it out in Puppeteer 1.12.2, which uses an earlier revision of chromium and that link test passes ok. I also had a look at I've pushed 5e0a0e7 to fix some of the preview tests. I noticed #14994 which causes one of the other preview tests to fail, but seems like a genuine issue that we should fix separately. |
b44e60b
to
5e0a0e7
Compare
Many thanks for submitting the fix 👍 I'm also surprised seeing that there are more test failures with popular plugins enabled 🤷♂️ |
5e0a0e7
to
804880c
Compare
This test should be fixed with c66f67f. |
I disabled WP Editor Meta Boxes test suite in 4cfb304. It should be further investigated separately as it seems to be fragile on its own. |
4cfb304
to
2adf348
Compare
I'm down to 2 failures locally:
|
I fixed the rich text test in e1334f0 and Puppeteer upgrade dbc196f resolved the other failure. However, there is another test with 2 failing snapshots which need to be investigated.
|
dbc196f
to
10153f0
Compare
There are still some random failures on Travis. https://travis-ci.com/WordPress/gutenberg/jobs/196309131#L1085
https://travis-ci.com/WordPress/gutenberg/jobs/196309137#L1083
|
https://travis-ci.com/WordPress/gutenberg/jobs/196309137#L1114
|
10153f0
to
aa4f816
Compare
This is what I have locally when I run
There are still some tests failing on Travis. All of them around applying formats, so there might be some issue in Chrome 75. /cc @ellatrix This is what I try now locally: for i in {1..20}; do npx wp-scripts test-e2e --config packages/e2e-tests/jest.config.js packages/e2e-tests/specs/writing-flow.test.js; done; It has failed 10 out of 20 times on one or both of the following tests:
|
3fd75b3
to
2632657
Compare
…avigation seems to behave differently. Simplifying and waiting for a selector seems to resolve the issue
2632657
to
2550add
Compare
Description
Upgrades
puppeteer
and related dependenciesjest-puppeteer
andexpect-puppeteer
to the latest versions.This is necessary to let Firefox folks to run our e2e tests with Firefox browser, see https://bugzilla.mozilla.org/show_bug.cgi?id=1539202 for more details :)
As of today, Gutenberg uses Puppeteer v1.6.1 which uses
Chromium 69.0.3494.0 (r575458)
. This PR updates Puppeteer to v1.17.0 which usesChromium 76.0.3803.0 (r662092)
. It's worth emphasizing that we officially support only 2 latest versions of Chrome.Disabled tests
I will add a more detailed list later...
Testing
npm run test-e2e
locally and on Travis