fix(cypress): Fix failing/flaky E2E tests #22460

codyml · 2022-12-20T00:31:25Z

SUMMARY

This PR attempts to fix some failing/flaky Cypress tests:

Horizontal FilterBar > should spot changes in "more filters" and apply their values (fail 23%, flake 22% on master last 7 days)
Drill to detail modal > Tier 1 charts > Modal actions > paginates (fail 8%, flake 30% on master last 7 days)
Charts list > Cross-referenced dashboards > should show the newly added dashboards in a tooltip (fail 0%, flake 80% on master last 7 days)
Horizontal FilterBar > should go from vertical to horizontal and the opposite (fail 3%, flake 22% on master last 7 days)
Horizontal FilterBar > should show all default actions in horizontal mode (fail 0%, flake 13% on master last 7 days)
Horizontal FilterBar > should display newly added filter (fail 2%, flake 13% on master last 7 days)

The code changes include:

Replacing .then with .should calls in drilltodetail.test.ts to allow for retries within the callback
Adding a timed .wait in nativeFilters.test.ts to give the filter bar orientation menu time to change orientation after scrolling into position
Switching the target for .type from the input's parent to the input itself in horizontal filter bar tests, as the input's parent was at times not considered "visible" by Cypress
Making Cypress wait for the confirmation toast to appear after saving a chart to a dashboard before navigating away from the page

TESTING INSTRUCTIONS

Check that E2E tests pass on CI, and feel free to re-run a few times to try to catch any extra issues
Run Cypress E2E tests locally several times with throttling enabled, specifically looking for issues in:
- dashboard/drilltodetail.test.ts
- dashboard/nativeFilters.test.ts
- chart_list/list.test.ts

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

…lters" and apply their values.'

codecov · 2022-12-20T01:24:13Z

Codecov Report

Merging #22460 (f1f0a91) into master (858796b) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #22460   +/-   ##
=======================================
  Coverage   66.89%   66.89%           
=======================================
  Files        1850     1850           
  Lines       70701    70701           
  Branches     7750     7750           
=======================================
  Hits        47292    47292           
  Misses      21393    21393           
  Partials     2016     2016

Flag	Coverage Δ
javascript	`53.86% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

villebro

LGTM, thanks for taking on this challenge, as CI has been pretty painful lately ❤️

villebro · 2022-12-20T10:11:50Z

superset-frontend/cypress-base/cypress/integration/dashboard/nativeFilters.test.ts

@@ -82,7 +82,7 @@ function prepareDashboardFilters(
  }).then(res => {
    const { body } = res;
    const dashboardId = body.result.id;
-    const allFilters: Record<string, any>[] = [];
+    const allFilters: Record<string, unknown>[] = [];


I believe these are Filters:

import { Filters } from '@superset-ui/core'; const allFilters: Record<string, Filters>[] = [];

Hmm, I tried this but it gave me a "Module '"@superset-ui/core"' has no exported member 'Filter'." error. I also didn't see anywhere else in superset-frontend/cypress-base that imports from @superset-ui/core – do you know if these packages are supposed to be isolated for any reason?

geido

LGTM!

kgabryje · 2022-12-20T12:56:05Z

superset-frontend/cypress-base/cypress/integration/dashboard/drilltodetail.test.ts

          expect($rows).to.contain('Amy');
        });
        // checking the paginated data
        cy.get('.ant-pagination-item')
          .should('have.length', 6)
-          .then($pages => {
+          .should($pages => {


Cool, I didn't know that should waits and then doesn't!

Me either, this PR was a learning experience!

kgabryje · 2022-12-20T13:00:25Z

superset-frontend/cypress-base/cypress/integration/dashboard/nativeFilters.test.ts

@@ -202,6 +202,7 @@ function openVerticalFilterBar() {

 function setFilterBarOrientation(orientation: 'vertical' | 'horizontal') {
  cy.getBySel('filterbar-orientation-icon').click();
+  cy.wait(250);


Can we add timeout to getBySel instead of this wait? Like cy.getBySel('dropdown-selectable-info', { timeout: 250 })

Actually, not sure which method do we prefer in Superset, so feel free to leave it as is 🙂

Nevermind, I didn't notice that it's merged 😄

Haha no worries! The issue seemed to be that the loading of the page caused some scrolling action while the menu was opening, which disrupted Cypress's ability to click on the menu item even though it is able to find it, so this .wait hopefully will let the scrolling finish in more cases before Cypress tries to click on the menu item.

EugeneTorap · 2022-12-21T10:37:22Z

Hi @codyml! I have problem with chart_list/filter.test.ts and dashboard_list/filter.test.ts

EugeneTorap · 2022-12-21T13:09:33Z

@codyml Can you see my #22489 PR and check failed cypress CI?

codyml added 5 commits December 19, 2022 21:55

Fix typing lint warning.

9757c7f

Adjust filter selection logic to fix 'should spot changes in "more fi…

4bdba37

…lters" and apply their values.'

Add wait to give dropdown time to reposition.

de8e46f

Change then to should to enable retries.

9b01795

Wait for chart save confirmation before proceeding.

f1f0a91

codyml requested review from jinghua-qa and geido as code owners December 20, 2022 00:31

pull-request-size bot added the size/S label Dec 20, 2022

villebro approved these changes Dec 20, 2022

View reviewed changes

geido approved these changes Dec 20, 2022

View reviewed changes

geido merged commit 21b56b2 into apache:master Dec 20, 2022

kgabryje reviewed Dec 20, 2022

View reviewed changes

This was referenced Jan 3, 2023

chore: Migrate /superset/search_queries to API v1 #22579

Merged

chore: Migrate /superset/queries/<last_updated_ms> to API v1 #22611

Merged

mistercrunch added the 🚢 2.1.3 label Feb 18, 2024

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 2.1.0 and removed 🚢 2.1.3 labels Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(cypress): Fix failing/flaky E2E tests #22460

fix(cypress): Fix failing/flaky E2E tests #22460

codyml commented Dec 20, 2022

codecov bot commented Dec 20, 2022

villebro left a comment

villebro Dec 20, 2022

codyml Dec 20, 2022 •

edited

Loading

geido left a comment

kgabryje Dec 20, 2022

codyml Dec 20, 2022

kgabryje Dec 20, 2022

kgabryje Dec 20, 2022

kgabryje Dec 20, 2022

codyml Dec 20, 2022 •

edited

Loading

EugeneTorap commented Dec 21, 2022

EugeneTorap commented Dec 21, 2022

fix(cypress): Fix failing/flaky E2E tests #22460

fix(cypress): Fix failing/flaky E2E tests #22460

Conversation

codyml commented Dec 20, 2022

SUMMARY

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

codecov bot commented Dec 20, 2022

Codecov Report

villebro left a comment

Choose a reason for hiding this comment

villebro Dec 20, 2022

Choose a reason for hiding this comment

codyml Dec 20, 2022 • edited Loading

Choose a reason for hiding this comment

geido left a comment

Choose a reason for hiding this comment

kgabryje Dec 20, 2022

Choose a reason for hiding this comment

codyml Dec 20, 2022

Choose a reason for hiding this comment

kgabryje Dec 20, 2022

Choose a reason for hiding this comment

kgabryje Dec 20, 2022

Choose a reason for hiding this comment

kgabryje Dec 20, 2022

Choose a reason for hiding this comment

codyml Dec 20, 2022 • edited Loading

Choose a reason for hiding this comment

EugeneTorap commented Dec 21, 2022

EugeneTorap commented Dec 21, 2022

codyml Dec 20, 2022 •

edited

Loading

codyml Dec 20, 2022 •

edited

Loading