Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gutenberg: Trigger E2E suite for v9.9.x upgrade #49599

Merged
merged 6 commits into from
Feb 12, 2021
Merged

Gutenberg: Trigger E2E suite for v9.9.x upgrade #49599

merged 6 commits into from
Feb 12, 2021

Conversation

WunderBart
Copy link
Member

@WunderBart WunderBart commented Feb 2, 2021 •
edited
Loading

Tracking issue: #49534.

Includes fix for failing tests where inserted block focus is stolen by the restore focus hook. See WordPress/gutenberg#28932 for more details.

ToDo:

A fix for the focus issue mentioned above has been already merged and will be included in the v10.0 release. This PR might need to be reverted when we upgrade to that version.

What about failing specs?

See the summary here: #49599 (comment)

@WunderBart WunderBart added [Type] Task DO NOT MERGE [Status] Needs e2e Testing Gutenberg Edge Run e2e tests with a special site that is running the latest Gutenberg plugin snapshot labels Feb 2, 2021
@matticbot
Copy link
Contributor

Test live: https://calypso.live/?branch=e2e/gutenberg-upgrade-v9.9

@WunderBart WunderBart mentioned this pull request Feb 2, 2021
Closed
46 tasks
@matticbot
Copy link
Contributor

matticbot commented Feb 2, 2021 •
edited
Loading

This PR does not affect the size of JS and CSS bundles shipped to the user's browser.

Generated by performance advisor bot at iscalypsofastyet.com.

@WunderBart WunderBart added [Status] Needs e2e Testing Gutenberg Edge Run e2e tests with a special site that is running the latest Gutenberg plugin snapshot and removed [Status] Needs e2e Testing Gutenberg Edge Run e2e tests with a special site that is running the latest Gutenberg plugin snapshot labels Feb 2, 2021
@WunderBart WunderBart added [Status] Needs e2e Testing Gutenberg Edge Run e2e tests with a special site that is running the latest Gutenberg plugin snapshot and removed [Status] Needs e2e Testing Gutenberg Edge Run e2e tests with a special site that is running the latest Gutenberg plugin snapshot labels Feb 4, 2021
@sirreal
Copy link
Member

sirreal commented Feb 5, 2021

Rebased and running with 9.9.0 stable on edge.

@sirreal sirreal added [Status] Needs e2e Testing Gutenberg Edge Run e2e tests with a special site that is running the latest Gutenberg plugin snapshot and removed [Status] Needs e2e Testing Gutenberg Edge Run e2e tests with a special site that is running the latest Gutenberg plugin snapshot labels Feb 5, 2021
@WunderBart
Copy link
Member Author

Rebased and running with 9.9.1 stable on edge.

@fullofcaffeine fullofcaffeine added [Status] Needs e2e Testing Gutenberg Edge Run e2e tests with a special site that is running the latest Gutenberg plugin snapshot and removed [Status] Needs e2e Testing Gutenberg Edge Run e2e tests with a special site that is running the latest Gutenberg plugin snapshot labels Feb 9, 2021
@fullofcaffeine fullofcaffeine added [Status] Needs e2e Testing Gutenberg Edge Run e2e tests with a special site that is running the latest Gutenberg plugin snapshot and removed [Status] Needs e2e Testing Gutenberg Edge Run e2e tests with a special site that is running the latest Gutenberg plugin snapshot labels Feb 10, 2021
@fullofcaffeine
Copy link
Contributor

Rebased and running with 9.9.2 stable on edge.

@fullofcaffeine fullofcaffeine force-pushed the e2e/gutenberg-upgrade-v9.9 branch 2 times, most recently from d7ad0c9 to 7fab6d4 Compare February 11, 2021 22:28
@fullofcaffeine fullofcaffeine changed the title Gutenberg: Trigger E2E suite for v9.9 upgrade Gutenberg: Trigger E2E suite for v9.9.x upgrade Feb 11, 2021
WunderBart
WunderBart commented Feb 12, 2021
View reviewed changes
test/e2e/lib/gutenberg/gutenberg-editor-component.js
initsWithChildFocus ? 'has-child-selected' : 'is-selected'
}[aria-label*='${ ariaLabel }']`;

let insertedBlockSelector = By.css( insertedBlockSelectorStr );
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the things I wanted to refactor at some point - the By interface has "locator strategies" that return locators, not selectors. A selector in this case is actually the string passed to the locator strategy (css() in this case). We use the selector naming for locators everywhere, so we should keep doing that to stay consistent, IMO. Just thought I'd mention the "locators" for future reference :)

https://www.selenium.dev/documentation/en/getting_started_with_webdriver/locating_elements/

@WunderBart
Copy link
Member Author

WunderBart commented Feb 12, 2021 •
edited
Loading

Because of this PR that just got merged (planned for v10.1), this might end up not being a workaround at all:

The first thing is that if you insert a block using the inserter, it's not "focused" anymore, which allows the inserter to stay open. From a screen reader perspective, this is correct. Moving the focus was the thing forcing us to actually close the inserter.

We need to keep changes from that PR in mind as they can cause more failed specs in our E2E GB suites.

edit

Looks like we have a fix already! :D WordPress/gutenberg#28962

@WunderBart
Copy link
Member Author

WunderBart commented Feb 12, 2021 •
edited
Loading

Summary of failing mobile specs:

✅ Currently, there are no failing mobile specs that run on Gutenberg edge sites. See details below.

Mailosaur-related (ignore, see p1612456065018600-slack-C1A1EKDGQ for context):

  • [WPCOM] Launch (mobile) @signup @parallel Launch a free site "before all" hook: Create a site as a new user for "Can launch a site"
  • [WPCOM] Launch (mobile) @signup @parallel Launch when having multiple sites "before all" hook: Create 2 free sites as a new user for "Can start launch flow and abandon"
  • [WPCOM] Launch (mobile) @signup @parallel Launch a free site "before all" hook: Create a site as a new user for "Can launch a site"
  • [WPCOM] Launch (mobile) @signup @parallel Launch when having multiple sites "before all" hook: Create 2 free sites as a new user for "Can start launch flow and abandon"

Non-Gutenberg-edge-related:

  • [WPCOM] Themes: All sites (mobile) Activate a theme @parallel when a theme more button is clicked when Activate is clicked Successful activation dialog should show the successful activation dialog
    • Doesn't seem to be flaky, failing on master as well.
  • [WPCOM] Sign Up (mobile, en) Sign up for a non-blog site on a premium paid plan through main flow using a coupon @signup @visdiff Can Remove Coupon
    • Flaky, passed locally.
  • [WPCOM] Plans: (mobile) Viewing a specific plan with coupon: @parallel @jetpack Remove any existing coupon
    • Flaky, passed locally.

Gutenberg-related:

  • [WPCOM] Calypso Gutenberg Editor: Posts (mobile) Insert embeds: @parallel Can publish and view content
    • Flaky, passed locally.

Summary of failing desktop specs:

✅ Currently, there are no failing desktop specs that run on Gutenberg edge sites. See details below.

Mailosaur-related (ignore, see p1612456065018600-slack-C1A1EKDGQ for context):

  • [WPCOM] Launch (desktop) @signup @parallel Launch a free site "before all" hook: Create a site as a new user for "Can launch a site"
  • [WPCOM] Launch (desktop) @signup @parallel Launch a free site "before all" hook: Create a site as a new user for "Can launch a site"
  • [WPCOM] Launch (desktop) @signup @parallel Launch when having multiple sites "before all" hook: Create 2 free sites as a new user for "Can start launch flow and abandon"
  • [WPCOM] Launch (desktop) @signup @parallel Launch when having multiple sites "before all" hook: Create 2 free sites as a new user for "Can start launch flow and abandon"
  • [WPCOM] Launch (desktop) @signup @parallel Launch a free site "before all" hook: Create a site as a new user for "Can launch a site"
  • [WPCOM] Launch (desktop) @signup @parallel Launch a free site "before all" hook: Create a site as a new user for "Can launch a site"
  • [WPCOM] Launch (desktop) @signup @parallel Launch when having multiple sites "before all" hook: Create 2 free sites as a new user for "Can start launch flow and abandon"
  • [WPCOM] Launch (desktop) @signup @parallel Launch when having multiple sites "before all" hook: Create 2 free sites as a new user for "Can start launch flow and abandon"

Non-Gutenberg-edge-related:

  • [WPCOM] Themes: All sites (desktop) Preview a theme @parallel when a theme more button is clicked when "Try & Customize" is clicked click try and customize popover
    • Doesn't seem to be flaky, failing on master as well.
  • [WPCOM] Themes: All sites (desktop) Activate a theme @parallel when a theme more button is clicked when Activate is clicked Successful activation dialog should show the successful activation dialog
    • Doesn't seem to be flaky, failing on master as well.
  • [WPCOM] Sign Up (desktop, en) Sign up for a non-blog site on a premium paid plan through main flow using a coupon @signup @visdiff Can Remove Coupon
    • Flaky, passed locally.
  • [WPCOM] Plans: (desktop) Viewing a specific plan with coupon: @parallel @jetpack Remove any existing coupon
    • Flaky, passed locally.

Overall, I don't see any specs that are related to Gutenberg v9.9. @fullofcaffeine's fix is working nicely, so I think we can ship this one and move on with the upgrade. cc @griffbrad

@WunderBart WunderBart marked this pull request as ready for review February 12, 2021 13:45
@matticbot matticbot added the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Feb 12, 2021
@fullofcaffeine fullofcaffeine merged commit 30f1c62 into trunk Feb 12, 2021
@fullofcaffeine fullofcaffeine deleted the e2e/gutenberg-upgrade-v9.9 branch February 12, 2021 16:08
@matticbot matticbot removed the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Feb 12, 2021
@WunderBart WunderBart mentioned this pull request Feb 15, 2021
Merged
fullofcaffeine pushed a commit that referenced this pull request Feb 15, 2021
@WunderBart @fullofcaffeine
Revert "Gutenberg: Trigger E2E suite for v9.9.x upgrade (#49599)"
6aaba61 
This reverts commit 30f1c62.
fullofcaffeine added a commit that referenced this pull request Feb 16, 2021
@WunderBart @fullofcaffeine
Gutenberg: Update the E2E suite to run correctly against v9.9.3 (#50092)
0c836eb 
* Revert "Gutenberg: Trigger E2E suite for v9.9.x upgrade (#49599)"

This reverts commit 30f1c62.

* Improve the main options menu selector

It fails in the mobile viewport because it matches the block context menu, which is returned instead.

* Close the welcome modal dialog before trying to select a page template in the mobile screen size

* Use clickIfPresent to close the welcome dialog, as it might not show in all scenarios

Co-authored-by: Marcelo Serpa <81248+fullofcaffeine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fullofcaffeine fullofcaffeine Awaiting requested review from fullofcaffeine

Assignees
No one assigned
Labels
[Status] Needs e2e Testing Gutenberg Edge Run e2e tests with a special site that is running the latest Gutenberg plugin snapshot [Type] Task
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@WunderBart @matticbot @sirreal @fullofcaffeine