Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large XHR response objects can exceed the maximum header size. #76

Closed
zachgibb opened this issue Oct 27, 2015 · 18 comments · Fixed by #4720
Closed

Large XHR response objects can exceed the maximum header size. #76

zachgibb opened this issue Oct 27, 2015 · 18 comments · Fixed by #4720

Comments

@zachgibb
Copy link

If I respond to route with a very large object, tests waiting for that response will fail with ERR_EMPTY_RESPONSE

cy
  .server()
  .route("POST", /route/, reallyLargeObject).as("willFail")
  .get(".submit").click()
  .wait("@willFail")
@brian-mann
Copy link
Member

Okay this is not a quick or easy fix.

Node sets a constant maximum total header size at 80kb for all headers. There is no way to get around this. We can't toss the data in the request body either, since that would be altering the entire request.

Probably the only way to do this is transfer the data over websockets while the server begins handling the request.

That means managing all kinds of state, but this would work. There are some other features of the XHR refactor that would also introduce websockets, so I'll hold off on implementing this until it can go in with some new architecture.

In the mean time you'll have to reduce the response sizes to less than 80kb, or just use a fixture and not an alias.

So just move reallyLargeObject to a fixture and update your route

cy.route("POST", /route/, "fixture:reallyLargeObject")

@acthp
Copy link

acthp commented Dec 7, 2017

Trying to find a workaround for this, e.g. manipulating the request, or the response, or the server object. The available hooks fire after the request is sent, and after the response is delivered to the app, so there's no opportunity to influence it. Also haven't found any way to access the server object, e.g. to override a method. Any suggestions? Using a fixture doesn't help, because it's fixed. ;)

@brian-mann
Copy link
Member

If your fixture exceeds the maximum headers you'll need to modify your app code, such as using a conditional if window.Cypress exists, and if so, to use a property on that (on the XHR response callback)

Then ahead of time in your test code, you can use a fixture and set that property on the Cypress object so your app code can pick it up.

@brian-mann
Copy link
Member

The key takeaway here is that Cypress is just javascript and it has native access to everything. You can share properties and data. You can collaborate between Cypress and your application code.

If your application code is "hardwired" to take a specific path, you can just modify it to open a seam to which you can use Cypress to modify that behavior.

@acthp
Copy link

acthp commented Dec 7, 2017

Just realized I could wrap onreadstatechange during onRequest, to flow the data into the app.

cy.route({
	url: '/api/bookmarks/bookmark*',
	method: 'GET',
	onRequest: xhr => {
		var orsc = xhr.xhr.onreadystatechange;
		xhr.xhr.onreadystatechange = function() {
			if (this.readyState === 4) {
				Object.defineProperty(this, 'response', {
					writable: true
				});
				this.response = content;
			}
			orsc.apply(this, arguments);
		};
	},
	response: 'placeholder',
});

The Cypress console shows response 'placeholder', but otherwise this functions as a workaround.

@brian-mann
Copy link
Member

Yup that would work.

You could also bind to onload instead of onreadystatechange. Would avoid the readyState conditional

@ankri
Copy link

ankri commented Feb 9, 2018

Thank you for your suggestions.

For guys coming from google. Here is our workaround:

put this in e.g. integration/utils/loadLargeFixture.js

export default function loadLargeFixture(url, response, method = 'GET') {
  return cy.route({
    url,
    method,
    onRequest: xhr => {
      const originalOnLoad = xhr.xhr.onload;
      xhr.xhr.onload = function() {
        Object.defineProperty(this, 'response', {
          writable: true
        });
        this.response = response;
        originalOnLoad.apply(this, xhr);
      };
    }
  });
}

Import the json and the loadLargeFixture function in your test:

import largeFixture from '../fixtures/veryLargeJson.json';
import loadLargeFixture from '../utils/loadLargeFixture';

describe('demo', () => {
  it('should work with large fixture', () => {
    cy.server();
    loadLargeFixture('/api/large', largeFixture).as('data');
    // do stuff
    cy.wait('@data')
    // do stuff
  });
});

@jennifer-shehane jennifer-shehane added the stage: ready for work The issue is reproducible and in scope label May 1, 2018
@jennifer-shehane
Copy link
Member

I spent some time stuck on this today. 😢 It wasn't very obvious because I made a change to my fixtures that must have JUST put it over size limit for responses. So working yesterday -> not working today.

The ERR_EMPTY_RESPONSE would not display on each failure either, instead if would just say:

CypressError: The network request for this XHR could not be made. Check your console for the reason.

I was able to inspect the network panel and see that the response had a size of 0 bytes, which clued me in on it being this issue.

screen shot 2018-05-01 at 3 19 02 pm

This only breaks when I generate a bunch of data for an array response (like to check pagination), so would be nice to have fixed.

@tolicodes
Copy link

tolicodes commented Aug 14, 2018

hah just found a super simple solution:
before I had

cy.fixture(`${endpoint}.json`).as(`fixture${endpoint}`);
cy.route(path, `@fixture${endpoint}.json`)
    .as(endpoint);

now:

cy.route(path, `fixture:${endpoint}.json`)
    .as(endpoint);

Cleaner and it works :)

@Kebie
Copy link

Kebie commented Oct 25, 2018

This error mixed with the no support for fetch() ruined my afternoon, and made me think I was crazy. Please add note in docs where it is suggested to use cy.fixture aliases when it seems to not be great for large jsons.

Just not using aliases like @tolicodes suggested fixed everything.

@saimonmoore
Copy link

I just ran into this today! @tolicodes method worked perfectly for me with a 1Mb fixture!

@cookiescrumbs
Copy link

cookiescrumbs commented Dec 5, 2018

@jennifer-shehane i'm getting the same CypressError: The network request for this XHR could not be made. Check your console for the reason.

Looking at the solution from @brian-mann at the top; I'm confused about the solution and why what I'm doing won't work... I'm not using an alias.

options = {
    method: 'POST',
    status: 200,
    url: '/playlist/search'
}

cy.fixture('playlist/search/playlist-search.dto.mock.json').then((data) => {
    options['response'] = data;
    cy.route(options);
});

Cheers

@AllanPinheiroDeLima
Copy link

AllanPinheiroDeLima commented Apr 5, 2019

@jennifer-shehane @brian-mann
I'm using an nedb implementation for simulate database inside my tests, so I have many custom functions running in and out all the time.

I've found something interesting, as the response sent through the server is meant to be in the body, why the response is sitting in the headers? Is there any reason to do that ? Because of this, the header gets much, much bigger than it needs

Is this a behavior or something else ?

i've attached a print.
Captura de tela de 2019-04-05 14-51-47

[EDIT]

I've managed to make the header less than 80kb and make it work again, but, still, can we make it work not by sending responses through the header ?

@nitzansan
Copy link

So I unfortunately and traumatically ran into this bug as well.
I was writing some new tests with new fixtures (programmatic fixtures and not static ones) and the cy.route + cy.wait failed with the ERR_EMPTY_RESPONSE error.

Unfortunately, I thought it was an issue on my side so I started to debug it and after rigorous testing found that the issue relates to big response size passed to cy.route.

As I mentioned before - Our fixtures are programmatic, which means that we don't have static json files holding fixtures and rather we have functions that we call and return as customizable fixtures. Therefore, @tolicodes solution doesn't seem to be relevant for us (unless I missed something about how to use cypress fixtures.
Additionally, I've tried using the loadLargeFixture function suggested by @ankri but it failed with XHR AssertionError.

  1. Is there any way to workaround this issue with my functional fixtures?
  2. I really think there should be an assertion internal in the cy.route implementation that throws a descriptive error when called with above-threshold size response.

@cypress-bot cypress-bot bot added stage: work in progress and removed stage: ready for work The issue is reproducible and in scope labels Aug 13, 2019
@mgurov
Copy link

mgurov commented Sep 1, 2019

Similarly to @nitzansan none of the workarounds with xhr events worked for my programmatic data.

What did work is a nasty hack of pre-creating a fixture programmatically:

cy.writeFile('cypress/fixtures/my_test_response.json', bigReponseObject);
cy.route('/whatever', 'fixture:my_test_response.json');

Can work as long as unique file names used and the fixtures saved this way are .gitignore'ed.

@cypress-bot cypress-bot bot added stage: needs review The PR code is done & tested, needs review and removed stage: work in progress labels Sep 12, 2019
brian-mann added a commit that referenced this issue Sep 24, 2019
* fix specs

* use debugger protocol for cookie handling in electron

* use latest gulp

* use rimraf instead of gulp-clean

* use electron 3.1.8 and node 10.2.1

* use gulp 4 in packages/static

* fix sendCommandAsync, log Schema.getDomains on CDP connect

* autofill e2e test name [skip ci]

* electron@5.0.7, see what new failures exist

* --no-sandbox for launching Electron

* update cookies logic for electron

* node 12

* update snapshot for new node

* update error message for new node

* stub sendCommandAsync

* only connect to socket if path has been replaced, fixes #4776

* update node-sass to support node 12

* skip wacky socket tests for now

* snapshot

* fix run_plugins_spec snapshot, don't include stack trace

* use --no-sandbox on linux to run as root

* allow sendCommandAsync to resolve

* use euid for root check

* log domains even if undefined

* don't worry about ending 1xx responses immediately anymore

* use --max-http-header-size, change max size from 8kb to 1mb, fix #76

* do not send 502 on failed websocket, just send back ECONNRESET

* update websocket spec port to not collide with other test

* update outdated expect

* Revert "only connect to socket if path has been replaced, fixes #4776"

This reverts commit f179eda.

* update gulp in root

* update https-proxy unit tests

* update network spec to properly close server

* update reporter spec

* update https-proxy-agent to fix node 10.10.0 change

discussion: https://github.com/nodejs/node/issues/24474\#issuecomment-511963799

* only pass --max-http-header-size on node >=12

* use own server-destroy implementation that supports secureConnect events

* oops

* update socket_spec

* electron 6.0.0

* console.table introduced in node 10

* change browserify entry to init.js

* handle edge case when no response body

* console.table added in node 10

* do not exit app when all BrowserWindows are closed

* update e2e snapshots

* value may not be null

* update plugins spec

* correct cookie expiry, use browser.getversion for CDP version check

* fix snapshotting for require stacks

* reorder cookies in spec

* warn when depreated electron callback apis are used

* only report 1 plugin error per process

* cleanup

* node 12.4.0, cypress/browsers:node12.4.0-chrome76 docker image

* update shell.openExternal to promisified

* update dialog.showOpenDialog to promisified

* update webContents.session.setProxy to promisified

* updating native dependencies since we don't need ancient node ABI support anymore

* WIP: switch cookies to simpler, jar-less approach

* WIP: switch cookies to simpler, jar-less approach

* making tests pass

* improve cookie filtering logic

* Remove unneeded Promise.try

* filter what makes it to the extension

* properly re-set superdomain cookies on cross-origin cy.visit

* allow comma-separated list of e2e tests

* sort cookies in order of expiration date, ascending

* updating tests, cleanup

* update tests

* version electron as a devDependency, electron@6.0.1

* cleanup, remove old automation

* cleanup, remove old automation

* bump chokidar to fix win10 + node12 issue

was seeing this on windows:
nuxt/nuxt#6035

fixed with version bump

* enable now-supported quit role, re-enable old tests

* don't need that arg there

* remove last deprecated callback electron invocations

* Delete cypress.json

* responding to PR feedback

* cleanup

* invoke

* use 'quit' role

* Use new appMenu role for Cypress menu on mac

* electron@6.0.2

* electron@6.0.3

* remove domain: cookie.domain and see what happens

* remove setErrorHandler

* Revert "remove domain: cookie.domain and see what happens"

This reverts commit 49e9168.

* add unit tests for cookies

* ci

* fix project-content css

* electron@6.0.4

* fix specs_list test

* electron@6.0.7

* some cleanup

* electron@6.0.9

* Update 8_reporters_spec.coffee.js

* electron@5.0.10 - Chromium 73, Node 12

* cli: fix the STDIN pipe on Windows (#5045)

* cli: pipe stdin

* uggh, here is the actual change

* update cli unit tests

* add unit test

* more permissive check for json to include application/vnd.api+j… (#5166)

* more permissive check for json to include

* add json test for content-type application/vnd.api+json

* cruder solution passes e2e tests locally, so let's go with that

* Remove 'charset' from content-type before checking if JSON

* fix eslint for fixture specs (#5176)

* update eslint to lint files within 'fixtures' in support files

- ignore some edge cases like jquery, jsx and obvious js files we wrote
with broken code

* Fixes from eslint to 'fixtures' files

* Catch env variable with reserved name CYPRESS_ENV 1621 (#1626)

* server: check CYPRESS_ENV variable when merging configs

* catch invalid CYPRESS_ENV value in CLI, close #1621

* linting

* sanitize platform in test snapshot

* linting

* update error message text

* add missing comma

* fix finally merge in JS code

* pass CLI linter

* fix log reference, should be debug

* use correct sinon reference

* update message, show first part in red

* update error message text

* Addresses #2953 (#5174)

* Addresses #2953

* Added proper test for new error message

* Didn't realize it ran this test as well, whoops

* Implementing changes as suggested by @jennifer-shehane

* Fixing tests and error output. Moved the checks to the start of the get command to ensure we always catch improper options

* Removing issue test since the querying spec covers it

* Using coffescript isArray check

* depromisify things that were promisified b/t electron 5 <=> 6

Revert "update shell.openExternal to promisified"

This reverts commit 8b6460d.

Revert "update dialog.showOpenDialog to promisified"

This reverts commit 5f178b0.

Revert "update webContents.session.setProxy to promisified"

This reverts commit 727df3a.

* node12.4.0-chrome76 => node12.0.0-chrome75

* fix tests for electron downgrade

* node12.0.0-chrome75 => node12.0.0-chrome73


Co-authored-by: Zach Bloomquist <github@chary.us>
Co-authored-by: Brian Mann <brian.mann86@gmail.com>
@cypress-bot cypress-bot bot added stage: pending release and removed stage: needs review The PR code is done & tested, needs review labels Sep 24, 2019
@cypress-bot
Copy link
Contributor

cypress-bot bot commented Sep 24, 2019

The code for this is done in cypress-io/cypress#4720, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

@cypress-io cypress-io deleted a comment from cypress-bot bot Oct 2, 2019
grabartley pushed a commit to grabartley/cypress that referenced this issue Oct 6, 2019
* fix specs

* use debugger protocol for cookie handling in electron

* use latest gulp

* use rimraf instead of gulp-clean

* use electron 3.1.8 and node 10.2.1

* use gulp 4 in packages/static

* fix sendCommandAsync, log Schema.getDomains on CDP connect

* autofill e2e test name [skip ci]

* electron@5.0.7, see what new failures exist

* --no-sandbox for launching Electron

* update cookies logic for electron

* node 12

* update snapshot for new node

* update error message for new node

* stub sendCommandAsync

* only connect to socket if path has been replaced, fixes cypress-io#4776

* update node-sass to support node 12

* skip wacky socket tests for now

* snapshot

* fix run_plugins_spec snapshot, don't include stack trace

* use --no-sandbox on linux to run as root

* allow sendCommandAsync to resolve

* use euid for root check

* log domains even if undefined

* don't worry about ending 1xx responses immediately anymore

* use --max-http-header-size, change max size from 8kb to 1mb, fix cypress-io#76

* do not send 502 on failed websocket, just send back ECONNRESET

* update websocket spec port to not collide with other test

* update outdated expect

* Revert "only connect to socket if path has been replaced, fixes cypress-io#4776"

This reverts commit f179eda.

* update gulp in root

* update https-proxy unit tests

* update network spec to properly close server

* update reporter spec

* update https-proxy-agent to fix node 10.10.0 change

discussion: https://github.com/nodejs/node/issues/24474\#issuecomment-511963799

* only pass --max-http-header-size on node >=12

* use own server-destroy implementation that supports secureConnect events

* oops

* update socket_spec

* electron 6.0.0

* console.table introduced in node 10

* change browserify entry to init.js

* handle edge case when no response body

* console.table added in node 10

* do not exit app when all BrowserWindows are closed

* update e2e snapshots

* value may not be null

* update plugins spec

* correct cookie expiry, use browser.getversion for CDP version check

* fix snapshotting for require stacks

* reorder cookies in spec

* warn when depreated electron callback apis are used

* only report 1 plugin error per process

* cleanup

* node 12.4.0, cypress/browsers:node12.4.0-chrome76 docker image

* update shell.openExternal to promisified

* update dialog.showOpenDialog to promisified

* update webContents.session.setProxy to promisified

* updating native dependencies since we don't need ancient node ABI support anymore

* WIP: switch cookies to simpler, jar-less approach

* WIP: switch cookies to simpler, jar-less approach

* making tests pass

* improve cookie filtering logic

* Remove unneeded Promise.try

* filter what makes it to the extension

* properly re-set superdomain cookies on cross-origin cy.visit

* allow comma-separated list of e2e tests

* sort cookies in order of expiration date, ascending

* updating tests, cleanup

* update tests

* version electron as a devDependency, electron@6.0.1

* cleanup, remove old automation

* cleanup, remove old automation

* bump chokidar to fix win10 + node12 issue

was seeing this on windows:
nuxt/nuxt#6035

fixed with version bump

* enable now-supported quit role, re-enable old tests

* don't need that arg there

* remove last deprecated callback electron invocations

* Delete cypress.json

* responding to PR feedback

* cleanup

* invoke

* use 'quit' role

* Use new appMenu role for Cypress menu on mac

* electron@6.0.2

* electron@6.0.3

* remove domain: cookie.domain and see what happens

* remove setErrorHandler

* Revert "remove domain: cookie.domain and see what happens"

This reverts commit 49e9168.

* add unit tests for cookies

* ci

* fix project-content css

* electron@6.0.4

* fix specs_list test

* electron@6.0.7

* some cleanup

* electron@6.0.9

* Update 8_reporters_spec.coffee.js

* electron@5.0.10 - Chromium 73, Node 12

* cli: fix the STDIN pipe on Windows (cypress-io#5045)

* cli: pipe stdin

* uggh, here is the actual change

* update cli unit tests

* add unit test

* more permissive check for json to include application/vnd.api+j… (cypress-io#5166)

* more permissive check for json to include

* add json test for content-type application/vnd.api+json

* cruder solution passes e2e tests locally, so let's go with that

* Remove 'charset' from content-type before checking if JSON

* fix eslint for fixture specs (cypress-io#5176)

* update eslint to lint files within 'fixtures' in support files

- ignore some edge cases like jquery, jsx and obvious js files we wrote
with broken code

* Fixes from eslint to 'fixtures' files

* Catch env variable with reserved name CYPRESS_ENV 1621 (cypress-io#1626)

* server: check CYPRESS_ENV variable when merging configs

* catch invalid CYPRESS_ENV value in CLI, close cypress-io#1621

* linting

* sanitize platform in test snapshot

* linting

* update error message text

* add missing comma

* fix finally merge in JS code

* pass CLI linter

* fix log reference, should be debug

* use correct sinon reference

* update message, show first part in red

* update error message text

* Addresses cypress-io#2953 (cypress-io#5174)

* Addresses cypress-io#2953

* Added proper test for new error message

* Didn't realize it ran this test as well, whoops

* Implementing changes as suggested by @jennifer-shehane

* Fixing tests and error output. Moved the checks to the start of the get command to ensure we always catch improper options

* Removing issue test since the querying spec covers it

* Using coffescript isArray check

* depromisify things that were promisified b/t electron 5 <=> 6

Revert "update shell.openExternal to promisified"

This reverts commit 8b6460d.

Revert "update dialog.showOpenDialog to promisified"

This reverts commit 5f178b0.

Revert "update webContents.session.setProxy to promisified"

This reverts commit 727df3a.

* node12.4.0-chrome76 => node12.0.0-chrome75

* fix tests for electron downgrade

* node12.0.0-chrome75 => node12.0.0-chrome73


Co-authored-by: Zach Bloomquist <github@chary.us>
Co-authored-by: Brian Mann <brian.mann86@gmail.com>
@GMchris
Copy link

GMchris commented Oct 17, 2019

The fixture: fix by @tolicodes gets my request to actually load, instead of timeout, however it still takes over a minute and a half for a 22MB response (I realise that's large, but our app requires it and we'll probably have even larger ones), which is frankly too slow when trying to run quite a few tests.

@cypress-bot
Copy link
Contributor

cypress-bot bot commented Oct 23, 2019

Released in 3.5.0.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet