Experimental feature: disableProxy

[csvan]

EDIT: A much better solution to this is suggested in this comment: #25201 (comment)

What would you like?

This has been raised in the past, but the original author gave up on it (and Cypress, unfortunately). Closed (but not resolved) issue is here for context: #22319 (comment)

This feature would add the experimental option to completely disable the Cypress proxy and let the running browser (and Cypress) speak directly to the network.

Why is this needed?

As explained in the docs, Cypress uses a proxy for, among other things, modifying code which potentially would break Cypress during runtime. However, the docs also acknowledge that not all sites are affected by such code, and even provides an option (modifyObstructiveCode) which partially disables it. The proxy is also used for several other features.

However, the proxy also incurs several notable penalties:

• No HTTP2/3 support (see especially Add HTTP/2 Support in Proxy Layer #3708 for implications)
• Only supports gzip compression (see Brotli encoding request fails (ERR_CONTENT_DECODING_FAILED) #6197)
• Unpredictable network behavior (outgoing headers modified etc)
• Most importantly - demonstrably severe performance impact, especially for modern web apps which use chunking extensively.

So while the proxy certainly has important uses, there should be an option to disable it completely if a team determines they have no use for the extra functionality and would like to optimise performance instead.

Other

While I advocate for Cypress personally at my workplace, I am seeing rising pressure from colleagues to consider Playwright instead - primarily due to performance reasons. The author of the original issue ended up doing the same.

I love Cypress approach to testing, but I can only acknowledge that performance is - and has been for years - one of the primary weaknesses of the framework. The team has done impressive work over the last releases (especially in 9 and 10) to address many sticking points, but much more needs to be done in order to level the playing field with competing solutions - especially Playwright. I believe this would be a step in the right direction.

[mike-plummer]

There is an in-flight PR to address a regression in the network proxy that may mitigate some of the performance concerns noted here: #25209

[lmiller1990]

I am also curious about this. My understanding is Cypress cannot function without some sort of network proxy. Not only do many features depend on it, but I (think) there's some fundamental requirement for some sort of network proxy (could be mistaken, but the concept of a network proxy has been in Cypress since the very early days, far before things like cy.intercept existed.

Whether we can make some of the features opt-in or opt-out would be another discussion.

Before doing this, since the goal is to go faster, there's a better alternative - updating our network stack to some more modern libraries, and using HTTP/2, which is up to 10x faster. I think this would help greatly with performance.

There is an issue tracking that here: #3708.

[adam-thomas-privitar]

I want to be able to do this, possibly with the ability to say which URLs are proxied and which are not.

If you try and run E2E tests locally against an application in dev mode that uses an unbundled dev tool like Vite, you have thousands of requests for static files. It takes about 20 seconds to load the app instead of < 0.5 seconds without cypress in play.

[csvan]

@lmiller1990 my understanding (and I am not too familiar with Cypress internals) is that the proxy is primarily necessary for:

1. Removing obstructive code (frame-busting etc)
2. Providing a layer for network stubbing and other forms of request-analytics

I am not sure either of the above are strictly necessary, and again not all projects need either of them. It would be helpful to know if there is anything in the proxy without Cypress cannot function altogether. Our projects have no intrusive code, for example, and we do not use any of the network features at all apart from visit and request.

[lmiller1990]

It would be helpful to know if there is anything in the proxy without Cypress cannot function altogether

This is what I am trying to find out. I'm sure there is something around how Cypress injects itself - whether we can decouple that and have some option to make the proxy do less and thus improve performance is another question.

@adam-thomas-privitar I think that's actually a bug that is fixed by #25209. You can test this out right now by installing the pre-release with that patch: 6b78454#comments. If you do try it, let me know if it helps. I also want to use Vite and I want to go fast.

Edit you tried it, thanks! I still think we can go faster, but if you see anything that seems like it's unusually slow, it's probably a bug - you should create an issue.

[csvan]

Awesome, many thanks for your efforts @lmiller1990 <3

[brian-mann]

Cypress will always need to work as a proxy in order to function correctly - however the performance problems you mentioned are absolutely critical for us to implement/fix. As @lmiller1990 and others have mentioned, we have already fixed a few known issues, but specifically HTTP/2 support is on our radar.

I'm not aware of how chunked responses could be impacted performance wise outside of HTTP/2 support. If that's something you could provide more clarity/examples of, we could take a look at it. There are an significant number of things we do under the hood in order to be performant... chunking for instance is definitely a main concern of ours.

/cc @flotwig

[csvan]

@brian-mann see e.g. #3708 (comment)

[adam-thomas-privitar]

Cypress will always need to work as a proxy in order to function correctly

I'm not doubting it, but I'm interested to know how cypress depends on it and what features it backs. I know not having it would mean opting out of certain functionality like requests appearing in logs, request mocking, framebusting etc. However, I would actually take that trade off, even if on "this is at your own risk and who knows what will/wont be supported in future" basis.

Is there a deeper dependency on it?

Perf is better in the latest cypress, but still notably slower in my vite use case. That use case results in what sounds like an edge-casey number of requests, but I suspect its common to develop against an unbundled dev tool since it makes fixing e2e tests easier.

[lmiller1990]

HTTP/2 support in Cypress will likely solve this issue due to its fully multiplexed nature. I'm not sure on the status of HTTP/2, I'll try and find out. I suspect this is why Vite is fast locally, but slow in Cypress; it's using HTTP/2 when you do local development via the Vite dev-server, but Cypress is not using HTTP/2.

[adam-thomas-privitar]

@lmiller1990 On my server, I am using Vite in the middleware mode in express instead of the Vite dev server CLI, and it's set up with http1. Without cypress, even on http1, it's incredibly fast. So I don't think that's the main reason Vite is fast in comparison. I think the main reason is the overhead of cypress handling the request for static assets, which are otherwise insanely fast when dealt with in a way that's just "read in a file and send response" (i.e. send it to the real server and dont do anything else). Most likely, if comparing the overhead of the request handling in Cypress versus if I even did have HTTP2 enabled -- you'd find the former value is an order of magnitude higher than the savings you can make on the latter.

Cypress is doing a lot of stuff with queues etc but it's all basically a no-op in the end for these assets and wasted computation. Direct to my HTTP1 server, with vite middleware (which is still node code at the end of the day so nothing special), it's rapid (can barely see it happen) over thousands of files. I reckon a fix for this, which doesn't involve removing proxying of requests altogether, probably could be to allow us to configure URL roots that should be immediately passed through with no extra work. I've considered patching this myself and building my own cypress to test it out, but not got around to it yet.

I feel this would satisfy the requirements since you would be opting out of Cypress functionality for specific URLs (note: not opting out of proxy, but opting out of the processing that is undertaken to provide the features) and not flipping on big global "I give up on all this cypress functionality" button. It feels like it'd be easier for Cypress team to maintain long term as well as you're not worrying about this subset of users for whom you can't even see the requests even if you wanted to. The canonical point of entry is maintained. Hopefully, this suggestion is worth considering. I can even help with implementation.

I'm not sure what all the queue stuff is about but my initial reaction when looking at it was "this is a lot of complexity". I don't know enough to comment on the value generally speaking, but I'd guess, on what are essentially static assets, it's probably not needed. You pretty much never want to log these requests or do anything with them in Cypress -- at least for unbundled stuff served from a dev server like this.

Whilst HTTP2 is always going to be beneficial and there are other good reasons for implementing it, it may also be optimizing the wrong thing to solve this particular problem. Sorry, I've only just realised this, it's actually pretty crucial info. Hopefully, at least if im correct, this post helps avoid wasted time!

[csvan]

reckon a fix for this, which doesn't involve removing proxying of requests altogether, probably could be to allow us to configure URL roots that should be immediately passed through with no extra work

This is a much better solution than my initial suggestion to disable the proxy altogether. It effectively subsumes it if you can simply pass * to whitelist all requests.

[lmiller1990]

Thanks for the info - I'll need some time to digest and think about this (and involve the right people internally, I'm far from an expert in this part of the code base). This investigation is certainly useful, especially considering a lot of work is happening in the proxy layer now - please keep an eye on this issue (and the HTTP/2 one) for more info as it comes.

[jennifer-shehane]

Performance is very complex and our team is working on adding better performance monitoring to track much of where time is being spent.

@adam-thomas-privitar You mentioned static assets being a thing that's causing a performance hit. Is this being observed during cypress open or cypress run? Are all of these XHRs being logged in the command log? I ask because you can turn off logging of XHR in the Command Log now since 12.8.0 using log: false as shown here, this has an opportunity to potentially improve performance.

[Narretz]

I think disableProxy could be really good for component testing. Intercepting requests shouldn't really be a thing in component tests anyway (and it's not really a thing for static resources in the first place)

Right now the vite bundler is basically unusable / doesn't provide any benefit over webpack, because all the indivdual js modules load so slow taken together that the better bundling performance is lost.

[lmiller1990]

disableProxy could be doable in CT, I think. I find cy.intercept useful in CT, though, but not everyone would be using the same workflow.

I am curious about the second part of your comment, though:

Right now the vite bundler is basically unusable / doesn't provide any benefit over webpack, because all the indivdual js modules load so slow taken together that the better bundling performance is lost.

Unusable

This implies it does not work. In this the case - eg, it cannot work in your project, or just emphasizing you find it slow?

doesn't provide any benefit over webpack, because all the indivdual js modules load so slow taken together that the better bundling performance is lost.

This is interesting to me - I've found in E2E testing Vite bundles are painfully slow due to that lack of HTTP2 support and multiplexing in the proxy layer, but this hasn't been an issue in Component Testing at all, since you are generally not loading an entire page, but just individual components.

Are your components tests slower with Vite as the bundler than webpack? Is this mainly locally or mainly on CI? I worked on migrating a huge code base recently to migrate from webpack to vite (100s of specs, 1000s of tests) and it was something like 4-5x faster in CI.

If you could share some benchmarks of webpack vs Vite that show it is slower, we can take a look and see what's going on.

[Narretz]

Hi @lmiller1990 thanks for taking the time to clarify!

just emphasizing you find it slow

Yeah I'm just saying it's very slow

This is interesting to me - I've found in E2E testing Vite bundles are painfully slow due to that lack of HTTP2 support and multiplexing in the proxy layer, but this hasn't been an issue in Component Testing at all, since you are generally not loading an entire page, but just individual components.

I think that's because of the difference between using Vite in dev mode and in build mode. For e2e, locally, you will usually use dev mode, where Vite creates an http/2 server that serves modules individually instead of bundling them. In CI, you will usually use build mode, i.e. bundled output. In the latter case, there should be almost no difference between vite and webpack.

I worked on migrating a huge code base recently to migrate from webpack to vite (100s of specs, 1000s of tests) and it was something like 4-5x faster in CI.

You mean Component Tests? Isn't that always vite dev mode with the http/2 server? And depending on the complexity of a component, this can still mean dozens of http requests. In my case it's as slow as webpack (5) or maybe even a little slower.

[lmiller1990]

Yeah, I meant component tests.

I think disableProxy could be really good for component testing

I'm assuming your entire comment is in the context of Component Testing - Cypress relies heavily on the proxy and End to End cannot function without it.

Using Vite (whether in E2E or CT) in dev mode will be have a performance penalty Cypress, regardless of Vite's dev mode using HTTP/2 - the reason is that everything goes through the Cypress proxy, which is not using HTTP/2 (yet), to my understanding. This causes a bottleneck.

One other solution, which is actually what we do, is yarn vite build --mode development. Our task is here. This works better for us, since we are able to End to End test against the latest code without using dev mode. This isn't really an answer to the "disableProxy" request, but it might be useful for some people.

As for disableProxy, I really have no idea how complex this would be -- I suspect it's an XY problem - the goal isn't really to disable the proxy, but to make Cypress faster. The best way to do that would likely be the HTTP/2 work, #3708, as discussed.

In the meantime, I wonder if the yarn vite build --watch --mode development technique is useful for End to End testing a Vite app?

[lennerd]

Question about the performance in component tests: what kind of operating system are you using?

We are seeing very slow test start up time (about 20 seconds until the Cypress actually starts testing) on Windows only. Imagine waiting for 20 seconds on every small change you are doing in your code. 😖 Unix systems seem to work fine though. Looking at the CLI output, it's visible the requests are handled (compared to UNIX equally) fast, but each requests for JavaScript assets are logged with small delays between each request.

Anybody can confirm this?

[lmiller1990]

@lennerd that sounds unusable, is this using webpack or Vite? I've found startup can be slow for webpack (especially with large bundles). I hope we can make startup faster. The actual feedback loop is generally really quick for me (1-2 seconds - however long webpack takes to recompile my test).

I wonder if there is a windows specific bottleneck. If someone can reproduce, we can definitely prioritize this - I've got a few big webpack projects, I can try take them for a spin on windows.

[lennerd]

Hey @lmiller1990, thanks for responding! 🙏

We use vite, which when used in development is serving each module/file separatly. This leads to huge amount of requests and thus a very long start-up time. Its really hard to debug so we only assuming at this point. Might be connected with CLI logging or the intercept proxy. 🤷‍♂️

[lmiller1990]

Just to confirm you see this issue with Component or End to End testing?

[lennerd]

We see this issue with component testing.

[lmiller1990]

Seems windows specific. I'm not sure if windows filesystem is just slow, but I don't see any windows specific code relating to Vite in our code base 🤔

[adam-thomas-privitar]

I notice come mention of HTTP2 above. As per #25201 (comment), there's a fair amount of evidence it's got nothing to do with it, unless its been disproven?

[csvan]

Possible workaround when using Chrome (along with more evidence that this has a real impact on real-life environments): #3708 (comment)

[pgross41]

Possible workaround when using Chrome (along with more evidence that this has a real impact on real-life environments): #3708 (comment)

Here are more detailed steps for implementing this workaround #22968 (comment)