Enable Lifecycle Events to be customizable

[brian-mann]

Why this is necessary

Cypress automatically executes code in "between" tests. The idea behind this is that Cypress will try to "reset the state" and enable each test to run in its own pure state without other tests affecting or leaking into one another.

Currently it does the following:

• clears localStorage
• clears sessionStorage
• clears cookies
• removes aliases you've created
• resets routes you've defined
• removes stubs, spies, clocks you've created
• clears the cache once before the browser opens

Currently what it does not do:

• clears the actual application under test

Why this needs to change

It really makes no sense the way we currently handle this. This is often a source of confusion we've seen from multiple users.

For instance it is:

• not clear that this is happening
• not clear how you can stop lifecycle events from applying
• not clear why we reset everything above but not the application itself (?)

Because we don't forcibly clear the application state it means that it "appears" nothing has changed between the tests, but under the hood everything that built up the current session was reset.

This often results with users clicking links only to find out they've been magically "logged out".

In addition to sessions being cleared, it is actually technically possible for your application to do things in the tiniest briefest moment that Cypress finishes running one test and "switches" to the new test. Since your application is still active in the window, that means it will continue to run everything it's built to run.

This can cause all kinds of havoc.

Here's a sequence of events that demonstrates the problem:

• before test 1
  • cypress resets state asynchronously
• test 1 begins
  • your cypress spec code runs
  • you visit your application
  • you setup cy.route to route XHR's
  • cypress finishes executing all commands
• test 1 finishes
  • cypress moves onto test 2
• before test 2
  • cypress resets state asynchronously
    • while this happens, the application you visited in test 1 runs a poll / timer which creates an XHR
    • the state from the routes has been reset, so instead of XHR being routed, it goes out to your server
    • your server throws a 500, and the onerror handler of your XHR bubbles up and changes your application's state
    • your application is now in an invalid state and you have no idea why
• test 2 begins
  • your cypress spec code runs
  • you don't visit your application again (you already did in test 1)
  • your commands fail because the state of your application in test 1 is now invalid
• test 2 finishes, displays error

The reason we don't hear every user complain about this - is that for one its incredibly rare for this to happen and represents a "perfect storm" of events.

Additionally when we originally built these lifecycle events, they were with the assumption that most users would be visiting their applications before each test.

When visiting before each test (and thus always recreating / rebuilding the sessions and state), you insulate yourself from all of these problems.

What users really want to do

With that said many of our users really want to do one thing: login once in a before hook during the first test, and then have all subsequent tests run with this session.

While we don't really officially endorse this as the best pattern - it often makes sense in these situations:

• your app loads slow (albeit you have a much bigger problem than writing tests)
• you're not actually mutating session state in tests and therefore theres no reason to enforce needing to have it cleared each time

Another example users have suggested is when you apply a "unit testing" pattern to integration tests and write a bunch of "tiny" tests with a single assertion. However, we consider this an anti-pattern.

Irrespective of what we believe - the bottom line is that only logging in once before your tests will without a doubt enable all of your tests to run faster. If they have less to do in each test, they will simply complete faster.

Therefore it's imperative that we adequately solve this use case, and in the documentation and guides we provide resources / suggestions / hints / tips / tricks / philosophy to guide you to making these choices.

BTW in case you're curious we believe most of the need for a single login in a before hook is mitigated completely by using programatic API's like cy.request for things like setting state and logging in. By avoiding your UI you skip most of the reasons tests are slow.

One thing we absolutely do not endorse and will never support is writing tests Edited by Jennifer Shehane: I have no idea what this was meant to originally say

Changes that need to be made

Clear the App

Enforce clearing the Application at the end of test if it is not the last test but before the next test runs any of its code.

In other words, unless this is the very last test, the moment test code finishes and before any async code runs we MUST blow the application out of the window so no async events, polls, timers, promises, etc, run.

This will 100% prevent application leakage in "between" the tests and ensure the application is torn down properly.

Update the GUI

Display a new "instrument section" that lists each lifecycle event that was applied (or skipped).

Create new commnad

We'll need a new cy.clearApp() command which brings into parity the other cy.clear* commands.

This enables you to issue it within a test as a "one-off" command the same way the others work.

Remove old API's

Deprecate / remove the Cypress.Cookies.preserveOnce API's since that would be superseded by these Lifecycle events.

Other Considerations

Having first class lifecycle events with their own programatic API also means we could turn these completely off when writing unit tests. As it stands Cypress automatically runs the clearing code in between each test. These are asynchronous and involve talking to the automation servers (in the browser and server) and therefore are quite slow (in terms of clock cycles).

Examples of how we could do this

I don't like the idea of making this global configuration in cypress.json. Instead I think we should create API's that enable you to control this in code.

Since Cypress has to know how you intend to apply lifecycle events this code would have to go before any tests are defined or run.

Likely this is what you want - you'd likely want these changes to propagate to an entire spec file.

// users_spec.js

Cypress.lifecycle() // returns an object with the current configuration

Cypress.lifecycle(false) // disable all lifecycle events
Cypress.lifecycle(true) // enable all lifecycle events

Cypress.lifecycle({
  clearApp: true, // leave this on
  clearInternals: true // leave this on
  clearCookies: false // nope
  clearLocalStorage: false // nope
  clearSessionStorage: false // nope
})

describe('my suite', () => {
  it('test 1', () => {
    ...
  })

  it('test 2', () => {
    ...
  })
})

Users wanting to apply Lifecycle events globally to all specs would just add the code into a support file.

Use Cases

At first glance it seems as if lifecycle events would be binary: you'd either want to have them all enabled (by default) or turn them all off.

Not so fast Jack - let's take a deeper look and why there's a bit more complexity than this:

Clear Everything (the default)

Pros

• Absolutely no state build up
• Simple, clear, insulated
• Promotes taking shortcuts and skipping the UI as much as possible

Cons

• Realistically a bit slower to run all tests

Summary

• Slowest but flake free. Easy to understand.

Clear Nothing (what many users want)

Pros

• You visit once in a before hook, usually establishing the session there
• All subsequent tests use this existing session and don't visit
• The tests will run faster

Cons

• All of the tests are loosely coupled to one another unless you're really careful
• The state from one test will immediately apply to the subsequent test
• When running a single test vs all you will likely see different behavior that is difficult to debug / understand
• You're still susceptible to actions in your application leaking into other tests (as described above in detail)

Summary

• Fastest but susceptible to edge cases, potential flake, and poorly written tests.

Clear Some Things (likely what you should do instead)

If you want the benefit of only establishing a session once, but don't want to problems of loosely coupled tests then you likely want this configuration:

Cypress.lifecycle(false) // disable everything
Cypress.lifecycle{
  clearApp: true // but always clear the app
})

describe('my suite', function () {
  before(function () {
    cy.login() // establish the session only once!
  })

  beforeEach(function () {
    cy.visit('/') // but continue to visit before each test
  })

  it('test 1')

  it('test 2')

  it('test 3')
})

Pros

• The current state of the app is always blown away before each test
• The session is preserved between tests avoiding this to be re-created

Cons

• Takes more explanation and deeper understanding of Cypress to know about this
• Requires visiting before each test since the app is blown away
• Keeping the same session around can lead to state issues on the server (if the server isn't stubbed)
• You're apt not to take shortcuts which means that the gains you receive from running all tests are lost when running and iterating on a single test

Summary

• Faster than the first open, slower than 2nd. Likely what you want instead.

[brian-mann]

Related to:

• Cypress should empty indexDB before onload #1383
• Localstorage cleared after first test #1349
• Allow reusing cy.server() in several tests and hooks #1183
• Add Cypress option to clear browser cache before each test #1124
• Unresolved API calls cause Electron to fail all subsequent tests #1055
• Whitelisted cookies preserved over runs, clearCookies() does not clear whitelisted cookies #808
• beforeunload when running Electron #796
• Problem cancelling XHRs from previous test #761
• Cypress Clearing Cookies During Test Execution #747
• Stubbed XHR that are triggered between 2 tests are not stubbed #538
• File watching does not execute actual code changes when site under test utilizes Service Workers. #702

[BenoitAverty]

One case where tests need to build up state between them is when each test is a step of a cucumber scenario. In that case, each test depends on the previous steps of the same scenario.

I'm using cypress-cucumber-preprocessor but I'm currently stuck because of this (I have opened badeball/cypress-cucumber-preprocessor#43 about this).

With the feaure discussed in this issues, it would be doable for the preprocessor to manage the state clearing in such a way that state would be cleared in between scenarios but not between steps of the same scenario.

I'm guessing this is still a long way down the road, but do you have an estimate on when you may be tackling this ?

[BenoitAverty]

In the meantime, if I wanted to experiment, can anyone point me to the direction of the function(s) or pieces of code that do the clearing ? I'd like to see if this can't be solved in the preprocessor itself.

[jennifer-shehane]

@iansjk 😂 I have no idea what that was meant to say. The original seems to reflect this same wording, but obviously this was missing some clause.

The Sessions API work is coming along and viewable here: #14350. The WIP documentation is here if you want to see what it enables you do test: cypress-io/cypress-documentation#3427

[dwilches]

Just arrived here searching for a way for Cypress to cache images and js files to speed up my tests.
Is there an estimated time for when this improvement will be done?

[vishwa-amit]

looking at the conversations from start to end .. doesn't seem to have a solution or workaround even after this has been discussed for a while :-(

[timini]

Any workaround available to stop cypress clearing localstorage between tests?

[GentryRiggen]

The workaround is easy but ugly. Only have one describe() => it() per spec test. You can use cy.log(message) to achieve a similar feel in the logs. Not ideal but gets around this problem.

[nabilamin]

@timini If you to want get rid of this functionality entirely you can use the following lines in Cypress' index.js file

before(() => {
    Cypress.LocalStorage.clear = function (keys) {
        // do something with the keys here
    };
});

This overwrites the function before your test runs.

[zekedvc]

Is there anyway I can delete browser data between test cases?

This is not working:

beforeEach(() => {
  cy.window().then((window) => {
    window.sessionStorage.clear();
    window.localStorage.clear();
  });
});

This also are not working:

cy.clearCookie()
cy.clearLocalStorage()
cy.clearCookies()

[vedraan]

I would really love to see an option to have cache cleared on every test re-run. Not having it is hindering use of Cypress as TDD tool.

If for example I'm working on a component which needs to GET something from the API, I would need to comment out that bit of test since after the first run the response status becomes 304 and the body is "". I know this is expected server/browser behavior and when just using Cypress as a test tool it's causing no issues. But as a developer I want to have the same option I'd normally have in my browser with dev tools open to disable all caching which would allow me to rerun the test with consistent results.

[DennisHuallanca]

Any solution?

[jennifer-shehane]

We released cy.session() as an experimental command (set experimentalSessionSupport to true in config) in Cypress 8.2.0. This command can be used to cache and restore cookies, localStorage, and sessionStorage. We especially see this command as being useful for testing Auth flows and recommend giving it a try to replace your existing tests around authentication and login.

The default behavior when using experimentalSessionSupport will also be to clear the entire state between tests, visiting about:blank.

If you have any feedback about cy.session() for general UX improvements, new features, or unexpected behavior, please leave your feedback in our Cypress Session Experimental Feedback Discussion.

[arildandreassen]

Is there a way to avoid clearing sessionStorage between each test? This is extremely cumbersome in our e2e suite.
We can override clearing the localStorage with Cypress.Localstorage.clear = (keys) = {}, but are there any ways to avoid clearing the sessionStorage the same way?

It would save a lot of time to not clear the session vs clearing and restoring it with cy.session().

[rezelute]

What are the recommended steps to stop a request spilling into the next test? This has caused me a lot of headaches particularly when tests run on CircleCI. The big issue is that I can never replicate it with Cypress open and only very few times with Cypress run in my local terminal.

Example: If at the end of it("test1"), i send a request to write "apple" into Textbox1. Then in it("test2"), the request can sometimes spill in here so I find that the Textbox contains "apple" before I even start to do anything with this test.

The way I have to currently get around it is by waiting at the end of every test for the request to be sent (and response comes back) before moving onto the next it("test2") test.

Any suggestions are really appreciated :)

[bahmutov]

See https://glebbahmutov.com/blog/visit-blank-page-between-tests/

…

On Thu, Nov 25, 2021 at 12:22 PM Hamzah ***@***.***> wrote: What are the recommended steps to stop a request spilling into the next test? This has caused me a lot of headaches particularly when tests run on CircleCI. The big issue is that I can never replicate it with Cypress open and only very few times with Cypress run in my local terminal. Example: If at the end of it("test1"), i send a request to write "apple" into Textbox1. Then in it("test2"), the request can sometimes spill in here so I find that the Textbox contains "apple" before I even start to do anything with this test. The way I have to currently get around it is by waiting at the end of every test for the request to be sent (and response comes back) before moving onto the next it("test2") test. Any suggestions are really appreciated :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#686 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ4BJVIV6PW3M6UTXDOVNTUNZWF3ANCNFSM4D4GU3FQ> .

-- Dr. Gleb Bahmutov, PhD Schedule video chat / phone call / meeting with me via https://calendly.com/bahmutov ***@***.*** @bahmutov ***@***.***> https://glebbahmutov.com/ https://glebbahmutov.com/blog https://github.com/bahmutov

[rezelute]

That's a really cool approach. Thank you for the time you have taken demonstrate the issue and the steps to solve it in the blog!
I do wonder though, why there isnt a better way that Cypress gives us to handle this issue... Maybe its intentional for purpose that I cant think of?

[csvan]

Like @dwilches et al, we would very much like to see the possibility to disable the browser cache clearing between each invocation of cypress run. E.g. if I have cached a number of static files in the browser during run A, I expect them to still be cached when I execute run B after it. Would that fall within the scope of this issue?

[quentinus95]

Hello, at this stage, is it possible to clear browser history using Cypress? This question is related to my now closed ticket here: #8219.

Thanks!

[shwarcu]

@jennifer-shehane @brian-mann is there any progress on that topic? I'm mostly interested in IndexedDB cleanup.
I know workarounds exist but I'm asking about native mechanisms in Cypress :)

[Gayathri-Kesavakannan]

See https://glebbahmutov.com/blog/visit-blank-page-between-tests/

On Thu, Nov 25, 2021 at 12:22 PM Hamzah @.***> wrote:

What are the recommended steps to stop a request spilling into the next
test? This has caused me a lot of headaches particularly when tests run on
CircleCI. The big issue is that I can never replicate it with Cypress open
and only very few times with Cypress run in my local terminal.

Example: If at the end of it("test1"), i send a request to write "apple"
into Textbox1. Then in it("test2"), the request can sometimes spill in here
so I find that the Textbox contains "apple" before I even start to do
anything with this test.

The way I have to currently get around it is by waiting at the end of
every test for the request to be sent (and response comes back) before
moving onto the next it("test2") test.

Any suggestions are really appreciated :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#686 (comment),
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAQ4BJVIV6PW3M6UTXDOVNTUNZWF3ANCNFSM4D4GU3FQ
.

--
Dr. Gleb Bahmutov, PhD

Schedule video chat / phone call / meeting with me via
https://calendly.com/bahmutov
@.*** @bahmutov @.***>
https://glebbahmutov.com/ https://glebbahmutov.com/blog
https://github.com/bahmutov

I am having issue #1055 , workaround is not working afterEach(() => {
cy.window().then((win) => {
win.location.href = 'about:blank'
})
Electron is still wait on 'about:blank' to load & fails in afterEach

Framework has login helper which is used throughout framework & multiple describe /context blocks per spec file. Any other workarounds ??

Thanks

[emilyrohrbough]

Closing this issue as done for what we are wanting to address at this time and tagging for the v12.0.0 release.

The main idea this issue captured was the idea of providing clearer insights into what Cypress handles for users between tests to ensure users write independent tests.

In v12.0.0 (soon-to-be-released), we are introducing the concept of Test Isolation. There will be two modes of test isolation, on and off, with on being the new default mode.

When test isolation is on, before each test, Cypress will:

• clear the page by visiting about:blank
• clear cookies
• clear local storage
• clear browser storage

This will prevent DOM and browser state bleed over between tests. The new cy.session() command can be leveraged to save and cache shared browser contexts.

If it is desired to persist DOM and browser state between a set of tests, you will be able to turn test isolation off for either the entire Cypress run or for the suite.

Also, in the upcoming v12.0.0 release, we are removing the old Cypress.Cookies.preserverOnce and Cypress.Cookies.defualts API.

The ideas outlined this issue that we are not current interested in implementing are:

• Surfacing lifecycle logs in the reporter
• providing a cy.clearApp() (i.e. clear page) command
• Providing granular control of what level of isolation can be configured to clear (or persist) between tests. Currently we are providing all or nothing with “on” and “off”. After this release if we receive feedback on why additional granularity it needed, we will open a new issue to consider enhancing this behavior.

[cypress-bot]

Released in 12.0.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v12.0.0, please open a new issue.