-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Interactivity API: Fix data-wp-on-document flaky test #58008
Conversation
This is odd. Locators just give the Playwright instructions on how to find an element. There should be no difference between defining a new locator with the same instructions and reusing an existing one. cc @kevin940726 |
|
Judging from the error reported:
It seems like it's correctly getting the element but the text is still |
It's possible. This is a Preact app. The event listener is attached using a await visibilityButton.click(); // State is updated
await expect(counter).toHaveText("1"); // DOM is updated
// `useEffect` runs here and register the event listener.
await page.keyboard.press("ArrowDown");
await expect(updatedCounter).toHaveText("2"); |
Since it's not a real scenario or a real app, I think we can insert an arbitrary timeout in between to wait for the JS to hydrate. WDYT? Something along these lines: await visibilityButton.click();
await expect( counter ).toHaveText( '1' );
+ // Wait for the JS to attach the event listener. (Ref the issue)
+ await page.evaluate( () => new Promise( ( resolve ) => requestAnimationFrame( resolve ) ) );
await page.keyboard.press( 'ArrowDown' );
await expect( counter ).toHaveText( '2' ); |
Added in 17227fd to test it out. But, is adding delays a common way to solve flaky tests?
EDIT: Oh, ok, that's not the problem. The problem is that the |
No, but this is a bit of an edge case here unfortunately 😅. Usually this kind of problem only happens with poor hydration performance. If we expect the event listeners to only attach after some user action, then there is going to be a delay in between. If the user is really fast (as fast as Playwright) then it's going to result in poor UX. Given that it's not a real app but only a test case, I think it's fine to add some workaround with comments. In practice, some use advanced tricks to "record" the events before the hydration and "replay" them after the app hydrates (similar to the GA scripts). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you!
I think this is a better solution as it doesn't require an arbitrary delay and relays on Playwright's internal waiting mechanisms: 1c82bb0 @kevin940726 let me know what you think. |
Size Change: +2.39 kB (0%) Total Size: 1.7 MB
ℹ️ View Unchanged
|
Let's merge this to see if it solves the issue. |
The GB bot reopened the same issue, but this time it is for @c4rl0sbr4v0 can you apply the same approach to see if that solves the issue? |
Warning: Type of PR label mismatch To merge this PR, it requires exactly 1 label indicating the type of PR. Other labels are optional and not being checked here.
Read more about Type labels in Gutenberg. Don't worry if you don't have the required permissions to add labels; the PR reviewer should be able to help with the task. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This approach works too! It's a little bit more work so I didn't want to block the PR 😅, but I'm all in for improving the tests! Thank you!
// Add the element back. | ||
await visibilityButton.click(); | ||
await expect( counter ).toHaveText( '1' ); | ||
await expect( counter ).toHaveText( '1' ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Duplicated assertions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, true!
@c4rl0sbr4v0 can you fix this in the PR that you make to solve this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in #58181
What?
Fixes #57983.
Why?
Nobody wants flaky tests.
How?
Defining a new location in playwright should wait for the locator to appear. (Which happens on the previously visibility button click.)