feat: experimental single tab run mode for component testing #23104

lmiller1990 · 2022-08-04T06:11:18Z

Resolves CT Performance Improvements: v10 slower than v9 #22353

User facing changelog

Introduce a new Cypress experiment, experimentalSingleTabRunMode, for Component Testing. All specs will execute in a single tab, similar to the Component Testing alpha in Cypress 9.

Additional details

Useful Linkd

Technical brief (Cypress team only)
Documentation Update
Binaries you can test

In this PR, I added an experimental flag that will reverted back to the Cypress 9 behavior, which will run all the specs for CT in a single tab. It's much faster (see the associated issue for numbers). There are some trade-offs, as noted in the technical brief, which is why this is behind an experimental flag.

This experiment was driven by feedback in #22353, and also from our dashboard users, noting running all CT specs was up to 50% slower.

Ideally, when we work on the Run All specs feature, we will explore and find out how to best isolate specs running in a single tab. This work will likely make this experiment obsolete, and we'd just deprecate it, in favor of the new run all specs feature. This is a significant project and unlikely to land anytime soon, thus this experiment.

Steps to test

pull this branch down
go in packages/launchpad, which enables the experiment
yarn cypress:run:ct --browser chrome --config video=false. 55s on my machine. ⚡
observe it's faster, and it runs all specs in 1 tab. You can add --headed to see it running in a single tab.
remove experimental flag from cypress.config.ts
yarn cypress:run:ct --browser chrome --config video=false. 91.9s on my machine. 🐌

How has the user experience changed?

Component Tests run much more quickly with this experiment enabled.

PR Tasks

Have tests been added/updated?
Has the original issue (or this PR, if no issue exists) been tagged with a release in ZenHub? (user-facing changes only)
Has a PR for user-facing changes been opened in cypress-documentation?
Have API changes been updated in the type definitions?

…al-single-tab-component-testing

cypress-bot · 2022-08-04T06:11:21Z

Thanks for taking the time to open a PR!

Create a Draft Pull Request if your PR is not ready for review. Mark the PR as Ready for Review when you're ready for a Cypress team member to review the PR.
Become familiar with the Code Review Checklist for guidelines on coding standards and what needs to be done before a PR can be merged.

cypress · 2022-08-04T06:29:46Z

Test summary

37828 • 0 • 615 • 0 • 7

Run details


Project	cypress
Status	Passed
Commit	`56bf522`
Started	Aug 16, 2022 12:05 AM
Ended	Aug 16, 2022 12:25 AM
Duration	20:24 💡
OS	Linux Debian - 11.3
Browser	Multiple

View run in Cypress Dashboard ➡️

Flakiness

	cypress/e2e/commands/net_stubbing.cy.ts		5
1	network stubbing > intercepting request > can delay with deprecated delayMs param
2	network stubbing > intercepting request > can delay with deprecated delayMs param
3	network stubbing > intercepting request > can delay with deprecated delayMs param
4	network stubbing > intercepting request > can delay with deprecated delayMs param
5	network stubbing > intercepting request > can delay with deprecated delayMs param
This comment includes only the first 5 flaky tests. See all 7 flaky tests in the Cypress Dashboard.

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

lmiller1990 · 2022-08-04T08:24:24Z

packages/server/lib/modes/run.js

@@ -1105,6 +1111,16 @@ module.exports = {
      return this.currentSetScreenshotMetadata(data)
    }

+    if (options.experimentalSingleTabRunMode && options.testingType === 'component' && !options.isFirstSpec) {
+      // reset browser state to match e2e behavior when opening/closing a new tab
+      return openProject.resetBrowserState().then(() => {


We reset the state here, much like we do between tabs, so this minimizes the chance of negatively impacting spec isolation.

maybe this doesn't matter since this is experimental, but if something goes wrong, this loses the test retries for CT.

Oh true, this seems like a problem - something we will definitely need to highlight in the notes, unless we can find a fix prior to releasing this experiment (if that's even acceptable, will need to update the tech brief with this drawback... good point).

Actually @emilyrohrbough I don't get it - I tried to use retries but it works okay: 4c99b4d#diff-ff7a203eea50ba8d34ac184c54310e07e4f279447a923ba9bec878e643b9e289R699-R700

Is there another way to configuring this or an edge case that I'm missing?

The retries are only for the initial browser launch right? I'm not sure we'd need retries for when we're navigating in this new way

…al-single-tab-component-testing

lmiller1990 · 2022-08-10T10:46:05Z

packages/app/src/runner/aut-iframe.ts

@@ -35,6 +35,14 @@ export class AutIframe {
    return $iframe
  }

+  destroy () {


This is new for single tab - we tear down the AUT between specs, right after we stop recording, but before we start the next spec.

For comparison of before/after, see: https://whimsical.com/spec-isolation-XiJyCyVUJmWQYRQbysgpP1

ryanthemanuel · 2022-08-10T14:26:57Z

packages/errors/src/errors.ts

+    return errTemplate`\
+        The ${fmt.highlight(`experimentalSingleTabRunMode`)} experiment is currently only supported for Component Testing.
+
+        If you are enjoying the experiment with Component Testing, please join the discussion here: http://on.cypress.io/experimental-single-tab-run-mode`


Maybe have a more open ended message? They may be enjoying it, but they may not be? Either way we would want feedback.

ryanthemanuel · 2022-08-10T14:35:13Z

packages/server/lib/modes/run.js

@@ -1218,6 +1234,12 @@ module.exports = {
        .catch(warnVideoRecordingFailed)
      }

+      const usingExperimentalSingleTabMode = testingType === 'component' && config.experimentalSingleTabRunMode
+
+      if (usingExperimentalSingleTabMode) {


Any reason we're doing this here rather than at line 1269? Seems like it might make sense to destroy the Aut at the same place that we were previously closing the tab.

Good point, I'll move it down near there, near where we close/change tab. e83e56c

astone123 · 2022-08-10T16:16:19Z

Running the CT specs in launchpad with this experiment went from ~46s to ~28s on my machine 👍🏻

lmiller1990 · 2022-08-11T03:07:29Z

@ryanthemanuel thanks for the quick review! Updated.

Did some manual QA on the video recording, seems to be working as expected.

emilyrohrbough · 2022-08-12T20:13:35Z

packages/app/src/runner/event-manager.ts

@@ -54,6 +55,8 @@ export class EventManager {
  studioRecorder: any
  selectorPlaygroundModel: any
  cypressInCypressMochaEvents: CypressInCypressMochaEvent[] = []
+  // Used for testing the experimentalSingleTabRunMode experiment. Ensures AUT is correctly destroyed between specs.
+  autDestroyedCount = 0


Any way we can setup this up in the e2e/ct support files so we don't have this hanging around in prod code?

Sure thing, nice idea. Done: 3cb9ab3

ryanthemanuel

Looks good. Nice work!

cypress-bot · 2022-08-16T20:50:52Z

Released in 10.6.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v10.6.0, please open a new issue.

brian-mann · 2022-08-23T15:14:33Z

@lmiller1990 can you upload a screenshot of what the settings UI looks like for E2E vs CT that indicate which experiments are available?

I would expect that experimentalSingleTabRunMode experiment option is displayed as being available to set in CT, but not in E2E. Additionally because this is a testingType specific option, I would expect the docs to explain this and the option to guide the user towards setting it in the right place (not the top level).

Currently the UI looks like this:

Notice that there isn't a place where it explains where to set the experimental flags. Since we now are building support for testingType specific options I would expect the UI to accommodate this.

lmiller1990 added 8 commits July 27, 2022 11:57

revive logic to run CT in a single tab

2ecdd91

add feature flag: experimentalSingleTabRunMode

7d52188

remove log

1b473df

Merge remote-tracking branch 'origin/develop' into lmiller/experiment…

918997e

…al-single-tab-component-testing

Merge remote-tracking branch 'origin/develop' into lmiller/experiment…

b5574b5

…al-single-tab-component-testing

reset browser state between tests

5c760d4

document single tab run mode experiment;

ff28d69

add system test for experimental run mode

48c637d

This was referenced Aug 4, 2022

chore: debugging CT performance slowness #22612

Closed

CT Performance Improvements: v10 slower than v9 #22353

Closed

lmiller1990 added 2 commits August 4, 2022 18:03

fix snapshots

7cf8da9

use more simple project for testing

3adcb40

lmiller1990 commented Aug 4, 2022

View reviewed changes

additional guard;

adfd7ce

lmiller1990 mentioned this pull request Aug 4, 2022

docs: document new experimentalSingleTabRunMode flag cypress-io/cypress-documentation#4638

Merged

fix test

ca5a16c

lmiller1990 marked this pull request as ready for review August 4, 2022 09:57

lmiller1990 requested review from a team as code owners August 4, 2022 09:57

lmiller1990 requested review from emilyrohrbough, mike-plummer, ryanthemanuel and a team and removed request for a team and mike-plummer August 4, 2022 09:57

lmiller1990 added 2 commits August 5, 2022 08:52

Merge remote-tracking branch 'origin/develop' into lmiller/experiment…

58efbac

…al-single-tab-component-testing

resolve conflicts

9861034

lmiller1990 commented Aug 10, 2022

View reviewed changes

build binaries take 2

293a0f4

lmiller1990 marked this pull request as ready for review August 10, 2022 11:27

lmiller1990 requested review from a team and brian-mann August 10, 2022 11:28

ryanthemanuel reviewed Aug 10, 2022

View reviewed changes

lmiller1990 added 3 commits August 11, 2022 10:40

make error message more open ended

d7aa638

destroy AUT later in run mode lifecycle

e83e56c

add additional assertion around experimental flag

bc430a9

lmiller1990 force-pushed the lmiller/experimental-single-tab-component-testing branch from f193fec to bc430a9 Compare August 11, 2022 00:49

simplify error

7453063

lmiller1990 mentioned this pull request Aug 11, 2022

v10.4.0 Component test React 18 warnings with mid sized apps #23169

Closed

emilyrohrbough reviewed Aug 12, 2022

View reviewed changes

lmiller1990 added 2 commits August 15, 2022 12:19

merge develop and resolve conficts

cb609e6

remove test code from production code

3cb9ab3

lmiller1990 force-pushed the lmiller/experimental-single-tab-component-testing branch from 0c9341e to 3cb9ab3 Compare August 15, 2022 02:45

merge in origin and resolve conflicts

785e1ae

ryanthemanuel approved these changes Aug 15, 2022

View reviewed changes

astone123 approved these changes Aug 15, 2022

View reviewed changes

emilyrohrbough approved these changes Aug 15, 2022

View reviewed changes

lmiller1990 added 2 commits August 16, 2022 09:57

merge in develop

80cd48a

fix conflict

56bf522

lmiller1990 merged commit 91beb90 into develop Aug 16, 2022

lmiller1990 deleted the lmiller/experimental-single-tab-component-testing branch August 16, 2022 00:44

cypress-bot bot locked as resolved and limited conversation to collaborators Aug 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: experimental single tab run mode for component testing #23104

feat: experimental single tab run mode for component testing #23104

lmiller1990 commented Aug 4, 2022 •

edited

Loading

cypress-bot bot commented Aug 4, 2022

cypress bot commented Aug 4, 2022 •

edited

Loading

lmiller1990 Aug 4, 2022

emilyrohrbough Aug 5, 2022

lmiller1990 Aug 8, 2022

lmiller1990 Aug 9, 2022

ryanthemanuel Aug 10, 2022

lmiller1990 Aug 10, 2022

ryanthemanuel Aug 10, 2022

lmiller1990 Aug 11, 2022

ryanthemanuel Aug 10, 2022

lmiller1990 Aug 11, 2022

astone123 commented Aug 10, 2022

lmiller1990 commented Aug 11, 2022 •

edited

Loading

emilyrohrbough Aug 12, 2022

lmiller1990 Aug 15, 2022

ryanthemanuel left a comment

cypress-bot bot commented Aug 16, 2022

brian-mann commented Aug 23, 2022 •

edited

Loading

feat: experimental single tab run mode for component testing #23104

feat: experimental single tab run mode for component testing #23104

Conversation

lmiller1990 commented Aug 4, 2022 • edited Loading

User facing changelog

Additional details

Useful Linkd

Steps to test

How has the user experience changed?

PR Tasks

cypress-bot bot commented Aug 4, 2022

cypress bot commented Aug 4, 2022 • edited Loading

Test summary

Run details

Flakiness

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

astone123 commented Aug 10, 2022

lmiller1990 commented Aug 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanthemanuel left a comment

Choose a reason for hiding this comment

cypress-bot bot commented Aug 16, 2022

brian-mann commented Aug 23, 2022 • edited Loading

lmiller1990 commented Aug 4, 2022 •

edited

Loading

cypress bot commented Aug 4, 2022 •

edited

Loading

lmiller1990 commented Aug 11, 2022 •

edited

Loading

brian-mann commented Aug 23, 2022 •

edited

Loading