Bad XHR performance with Google Cloud Firestore. Like really bad.

[bdiz]

Current behavior:

When making requests to Firestore outside of Cypress, XHR responses take less that a couple seconds (at most).
When making requests to Firestore inside of Cypress, XHR responses take more than 60 seconds.

Desired behavior:

When not stubbing, I expect XHRs to have the same performance inside of a Cypress spec as they do when my site is hosted or when using a local server.

Steps to reproduce:

I created an example app which will hit the actual Firestore server. It includes a button to send an XHR and a timer which will run until the response is returned.

git clone https://github.com/bdiz/cypress-firestore-performance.git
cd cypress-firestore-performance
npm install
npm run serve
# click the "Add document to Firestore" button and observe the XHR finish within a couple seconds
# CTRL-C to kill the server
npm run test:e2e
# Run the test.js spec and click the "Add document to Firestore" button and observe the XHR finishes in 30 seconds plus.

Versions

Cypress 3.1.0
Ubuntu 16.10
Chrome 67 (used for hitting local server and inside test runner)

[leyenda]

Issue #1150 refers to this.

[orzarchi]

I can confirm this still doesn't work in the latest develop branch.
If anyone requires help reproducing let me know, but I think simple running any kind of firestore query will do the trick - the query never completes.
Hopefully this will be get looked at soon, as we our company will soon be forced to choose another testing solution :(

[pvdyck]

It is working for me, no more delay on firestore queries.
So yes, it could be useful to provide a sample repo with failing tests ?

[lilaconlee]

We're still able to replicate the issue with the repo that was originally provided and with the Firestore quickstart project, but still need some time to figure out a fix.

It's in a queue with a couple of other high priority, but pretty time consuming issues that we'll get to as soon as possible.

[afcastano]

Thank you for looking at this and for this awesome product.
I really want to use it but our web app is heavily using Firestore. Any idea when this could be fixed? I need to decide soon which testing framework should we use... :(

Thanks again =)

[fredrik-sogaard]

The latest 3.1.1 release seems to have fixed this for us.

[afcastano]

@fredrik-sogaard Where can I get 3.1.1? Is that develop branch?

I cloned develop and ran it using npm start.
Still getting the same error...

[fredrik-sogaard]

It's on npm: https://www.npmjs.com/package/cypress

[afcastano]

@fredrik-sogaard duh... didn't see the new release... anyway... version 3.1.1 still has the problem for me... :(

[emistorrs]

Any update on this? My company's project is entirely serviced by firestore, and we were really hoping to use cypress to do our integration tests.

[travwritescode]

This issue has been under development on and off for a year now. Is there any update on deliverability? My team is testing two apps that are dependent on Firebase and we would really like for Cypress to be the tool we use for these.

[Rednegniw]

I second this. Firestore requests have really big delay with Cypress for me.

[prescottprue]

I have experienced this only periodically in applications like fireadmin.io (here are the tests). The tests for the larger application we run at the company I work for and a number of folks using cypress-firebase have mentioned they see this issue very regularly

[prescottprue]

For those looking for a repro of not being able to see Firestore emulator in Cypress, which seems to be similar/related, I provide one in this issue: #6350.

In an issue opened in firebase-tools, it was mentioned that passing experimentalForceLongPolling: true to firestore settings that seems help. When setting this in the above mentioned repro, the data loads!

There was also the following theory from @wvanderdeijl that seems to make sense:

I think the underlying problem is that Cypress is intercepting all network traffic so it can monitor and sometimes mock. However, the webchannel protocol used by firestore has multiple replies over the same http request. The Cypress code cannot handle this and will only forward the first reply and ignore the rest.

^ Also aligns with what @mslooten mentioned above (so it seems that it isn't specific to AngularFire, but instead just Firestore in general)

[wilhuff]

I work on Firestore, so I can confirm: the default behavior of Firestore's web SDK is to make use of WebChannel's streaming mode. The client makes what looks like an XHR, but then the server will hold the response open for 60 seconds and send as many server-initiated responses as it can during that time window.

The experimentalForLongPolling option forces the server to send only a single response per request. While this works around issues where proxies buffer responses, it also introduces a 1/2 RTT delay between every response. You won't notice this developing your app in the US talking to a Firestore instance in us-central, but RTT from many parts of the world to the same instance is over 250 ms. Don't just blanket enable this option if you can avoid it.

[mesqueeb]

I have this issue as well.

Doing

  if (window.Cypress) {
        Firebase.firestore().settings({ experimentalForceLongPolling: true })
    }

solved it

[RaviH]

Thank you @mesqueeb for an example setup. Thank you @wilhub for confirming it; confirmation from someone close to the code helps. Is there any thoughts from cypress team about working with/around this? Firestore may remove this flag as it's experimental; besides would it be possible for Cypress to handle it auto-magically without user googling around for the problem.

[TDAK1509]

To only enabled in localhost, maybe you can do this

const db = firebase.firestore();

if (location.hostname === "localhost") {
  db.settings({
    experimentalForceLongPolling: true,
  });
}

[jwmcelr]

We are using the experimentalForceLongPolling: true option. It makes our cypress tests work with Firebase/Firestore, but it is still extremely slow and unreliable (in order to get many things to work consistently we have to set extremely long timeouts - sometimes as high as 60 or 90 seconds). We are using Firestore realtime updates to trigger things to happen in the app, so I don't know if that causes additional slowness you might not see with direct Firestore queries.

[yglin]

FYI, with firebase version 9 you have to use initialFirestore to opt-in the settings, like so
const firestore = initializeFirestore(app, { experimentalForceLongPolling: true })
And this must be called before any further access to firestore, including connectFirestoreEmulator(firestore, host, port)

[andershoegh]

Any plans for when to address the mentioned performance issues?

[ateirney]

In case it is useful for other people to know, one approach I have taken is to avoid the cypress proxy. An example of bypassing the cypress proxy for chrome is as follows.

  on("before:browser:launch", (browser = {}, launchOptions) => {
    if (browser.name === "chrome") {
      launchOptions.args.push("--proxy-bypass-list=<-loopback>,firestore.googleapis.com")
    }
    return launchOptions;
  })

[sceee]

My feeling is that the test reliability regarding this issue became much worse starting with cypress 8.6.0 - until v8.5.0 the performance with experimentalForceLongPolling was okay, but since v8.6.0 the tests become very unreliable.

[greegus]

Also experiencing the same issue – we are still getting Failed to get document because the client is offline within our Cypress tests, even with the experimentalForceLongPolling or experimentalAutoDetectLongPolling flag set in the compiled app using emulators (firebase 9.6.3, cypress 9.5.1). Outside Cypress, the app runs just fine.

@andrew-teirney also tried to set the proposed proxy-bypass-list, but without any results :/

[jwmcelr]

I believe I've partially worked through this using the comment above by @andrew-teirney. That comment didn't exactly do the trick for me, but it was close, and it led me to greatly increase my understanding of how Cypress is working. And with it I was able to comment out the experimentalForceLongPolling:true option that was causing Cypress to be so slow and unreliable for us while using Firebase/Firestore. I'll try to explain...

This is the code I put in the cypress/plugins/index.js file:

on('before:browser:launch', (browser = {}, launchOptions) => {
    if (browser.name === 'chrome') {
      // Add localhost:8080, the Firestore emulator host:port when running locally, to the Chrome proxy bypass
      // So Cypress doesn't jack with it
      launchOptions.args.push('--proxy-bypass-list=<-loopback>,localhost:8080');
      /*
      // Options that provide marginal memory improvement in Chrome, but seem to break headed mode
      // Leaving them out for now as I don't fully understand them and the improvement is marginal
      launchOptions.args.push('--ChromeOSMemoryPressureHandling');
      launchOptions.args.push('--renderer-process-limit=1');
      launchOptions.args.push('--single-process');
      launchOptions.args.push('--disable-dev-shm-usage');
      */
    }
    return launchOptions;
  });

First, these are chrome options that get passed to Chrome when Cypress starts it, so they only apply if the browser you're using for the test is Chrome. If you do a search for chrome options you can pass in, you'll see that proxy-bypass-list is telling Chome "hey, that proxy I told you about, don't send things for these hosts/hosts:ports through it." But you notice I didn't set a proxy, and the Chrome docs say this option only takes effect if you set a proxy-server. So where is the proxy coming from? Cypress. If you console.log(JSON.stringify(launchOptions)), you'll see there is a localhost:someweirdport proxy being passed in and if you open your browser and go to it, it's Cypress. So I think this is how Cypress is intercepting everything so you can mock it, etc. By passing in this "proxy-bypass-list" we're preventing Cypress from intercepting certain things, which is what we want to do for Firestore (localhost:8080 is the port we have our Firestore emulator running on).

Since by default running cypress in headed mode runs in Chrome, passing the localhost:8080 to chrome in the proxy-bypass-list was enough to allow me to comment out experimentalForceLongPolling:true, and the tests in headed mode worked immediately. Which was awesome because I could never get them to even partially work before without experimentalForceLongPolling:true.

By default Cypress runs Electron in headless mode though, so our automated tests would still fail without experimentalForceLongPolling:true, just as they always had. So I passed the --browser chrome option into the cypress run command in our package.json to tell Cypress to run against Chrome in headless mode too, and voila, it worked there too!

But now there was a big problem when running in our CI/CD pipeline. We use cucumber, and any cucumber feature file with more than a couple of tests would fail. It would work fine on my machine in headless mode, but fail in the CI/CD pipeline (we're limited to 8 gigs in our CI/CD pipeline VM's). I added the DEBUG=cypress:* flag to be before the cypress run command in package.json (we're using cross-env, not sure if this works without that or if you'd have to set this variable manually in a terminal). Adding that Debug flag prints out the memory being used by Cypress periodically in a nice chart, and breaks it down between the browser (Chrome in this case) and other things being spun up by Cypress. And boy is Chrome a memory hog, which is what was causing any Cucumber feature file with more than a few tests to fail by causing out of memory errors in our CI/CD VM's.

So then I thought maybe I could go back to Electron and pass in a similar proxy-bypass-filter option to Electron. But although there is some documentation saying you can do this with an environment variable (here and here), I couldn't get it to work. I tried setting it in package.json as part of the cross-env command and also as a straight up terminal variable before I ran the package.json command, and it just didn't seem to take effect. It wasn't the only thing: for that DEBUG=cypress:* flag I mentioned above you can pass in options to debug specific things (like memory), but that never seemed to take effect either - the only way it would work is with DEBUG=cypress:*, which prints out every debug message under the sun.

So for now we are left splitting our feature files that are causing out of memory issues on our CI/CD pipeline VM's into multiple feature files. Apparently after every feature file cypress-cucumber resets the browser, clearing the memory. Sometimes we're having to split a file with 2 tests into 2 files with 1 test each. But that seems a small price to pay to have the experimentalForceLongPolling option disabled. Our tests are reliably passing, and run so much faster now we're finding things that were flaky we'd never noticed before because our tests ran so ridiculously slow.

I'm hoping this is just a first step. I'd like to put that DEBUG option back on with Electron and the experimentalForceLongPolling turned back on to see how much memory Electron is consuming compared to Chrome, because we never had these memory issues with Electron. And I'm hoping now that I've gotten this far, Cypress may be able to provide some guidance about how to get passing the proxy bypass into Electron to work.