CT Performance Improvements: v10 slower than v9 #22353
Comments
cypress-bot
bot
added
the
stage: investigating
|
We're seeing a 3x increase in more complex spec files (from 1651ms to 4887ms) after upgrading to v10 ( |
|
To make sure, that problem is not on our side, I have cloned your example from https://github.com/cypress-io/cypress-component-testing-apps/tree/main/react-webpack5-js and I think it has the same problem.
But the whole process of |
|
I've not fully investigated, but our tests suite went from taking around 14 minutes up to 19 minutes since upgrading to 10.1.0 from 9.7.0 |
|
yeah, we also experience tests to be slower than in v9 |
|
@TomasSlama thank you for the report, and thanks all for the corroborating comments. I have the |
|
There is definitely a slowdown here. Running your
It appears that with 10.x we do some cleanup work/reset some browser state between spec executions that is the bulk of the added time. This time increases linearly with the number of specs included with a given run. The debug log output from the 10.1.0 run is provided below. Thanks again for the report. I'm going to direct this issue to the CT team to investigate mitigating factors. 10.1.0 debug output between spec run start and actually executing tests in the spec (click me) |
cypress-bot
bot
added
stage: routed to ct
and removed
stage: investigating
tbiethman
added
CT
|
Same here. Almost double as slow. On v9 my 200+ component tests would run on Github Actions in 8m 24s Sometimes it even goes over 20min, so I had to increase the timeout from 20min to 30min for now... I am using |
|
Maybe dup/related? #22208 I am looking into this. Performance is key, fixing this is the highest priority. I'll put my notes here. I ran these on the branches Cypress 9 - 175s. Cypress 9 CT: Average time per spec in ms 859ms There is a delta between my two repos of 51 seconds - so, we spent 51 seconds doing (some extra thing) in Cypress 10 for component testing. I will find out what. About 12 of those seconds are accounted for in the actual test spec execution, and the remaining 39 in (something). I'm guessing the actual test execution time is likely to be identical, since we inject the same code (your test and components). The ~125ms is likely also caused by whatever is causing the major slowdown. Edit: Some additional logging from Cypress 10 we don't have in 9, might unearth some changes. I'm kind of thinking out loud here. One change in Cypress 10 for component testing is a new tab per spec, to better improve isolation between specs. E2E always did this - in fact, it opened an entire browser per spec, so this pref regression would be only noticable for component testing. Looking at some logs, the main things that stick out: I'll dive deeper, but I'm pretty sure the main cost is the new tab per spec, which also means we have some overhead with
I'm not sure what resetting a project instance does exactly, I'll find out. Also, we use GraphQL in the open mode UI, but not run mode - it should not be starting or even making an appearance, but I think we have a web socket that is erronously starting up. Lots of logs## Cypress 9We proceed from Cypress 10: We proceed from We have this additional logging in between: Full log: Between Specs LogsCypress 9 Running: App_91.cy.jsx (2 of 2)
cypress:server:run about to run spec { spec: { name: 'App_91.cy.jsx', relative: 'src/App_91.cy.jsx', absolute: '/home/lachlan/code/dump/lots-of-specs-9/src/App_91.cy.jsx', specType: 'component' }, isHeadless: true, browser: { name: 'chrome', family: 'chromium', channel: 'stable', displayName: 'Chrome', version: '103.0.5060.53', path: 'google-chrome', minSupportedVersion: 64, majorVersion: 103, isHeadless: true, isHeaded: false } } +1ms
cypress:server:plugins plugin event registered? { event: 'before:spec', isRegistered: false } +417ms
cypress:server:run video recording has been enabled. video: true +0ms
cypress:server:video capture started { command: "ffmpeg -n 20 /home/lachlan/.cache/Cypress/9.7.0/Cypress/resources/app/node_modules/@ffmpeg-installer/linux-x64/ffmpeg -f image2pipe -use_wallclock_as_timestamps 1 -i pipe:0 -y -vcodec libx264 -filter:v crop='floor(in_w/2)*2:floor(in_h/2)*2' -preset ultrafast /home/lachlan/code/dump/lots-of-specs-9/cypress/videos/App_91.cy.jsx.mp4" } +18ms
cypress:server:run waiting for socket to connect and browser to launch... +12ms
cypress:server:run navigating to next spec +1ms
cypress:server:project_utils get spec url: /home/lachlan/code/dump/lots-of-specs-9/src/App_91.cy.jsx for spec type component +4s
cypress:server:project_utils prefixed path for spec { pathToSpec: '/home/lachlan/code/dump/lots-of-specs-9/src/App_91.cy.jsx', type: 'component', url: '/component/App_91.cy.jsx' } +0ms
cypress:server:project_utils return path to spec { specType: 'component', absoluteSpecPath: '/home/lachlan/code/dump/lots-of-specs-9/src/App_91.cy.jsx', prefixedPath: '/component/App_91.cy.jsx', url: 'http://localhost:8080/__/#/tests/component/App_91.cy.jsx' } +0ms
cypress:server:video capture stderr log { message: 'ffmpeg version N-47683-g0e8eb07980-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2018 the FFmpeg developers' } +1ms
cypress:server:video capture stderr log { message: ' built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516' } +1ms
cypress:server:video capture stderr log { message: ' configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg' } +0ms
cypress:server:video capture stderr log { message: ' libavutil 56. 24.101 / 56. 24.101' } +0ms
cypress:server:video capture stderr log { message: ' libavcodec 58. 42.100 / 58. 42.100' } +0ms
cypress:server:video capture stderr log { message: ' libavformat 58. 24.100 / 58. 24.100' } +0ms
cypress:server:video capture stderr log { message: ' libavdevice 58. 6.101 / 58. 6.101' } +0ms
cypress:server:video capture stderr log { message: ' libavfilter 7. 46.101 / 7. 46.101' } +0ms
cypress:server:video capture stderr log { message: ' libswscale 5. 4.100 / 5. 4.100' } +0ms
cypress:server:video capture stderr log { message: ' libswresample 3. 4.100 / 3. 4.100' } +0ms
cypress:server:video capture stderr log { message: ' libpostproc 55. 4.100 / 55. 4.100' } +0ms
cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 53722 } +3s
cypress:server:server-base Got CONNECT request from localhost:8080 +3s
cypress:https-proxy Writing browserSocket connection headers { url: 'localhost:8080', headLength: 0, headers: { host: 'localhost:8080', 'proxy-connection': 'keep-alive', 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/103.0.5060.53 Safari/537.36' } } +3s
cypress:https-proxy Got first head bytes { url: 'localhost:8080', head: 'GET /ws HTTP/1.1\r\nHost: localhost:8080\r\nConnection: Upgrade\r\nPra' } +0ms
cypress:https-proxy Making intercepted connection to 41055 +0ms
cypress:network:connect successfully connected { opts: { port: 41055, host: 'localhost', getDelayMsForRetry: [Function: getDelayForRetry] }, iteration: 0 } +3s
cypress:https-proxy received upstreamSocket callback for request { port: 41055, hostname: 'localhost', err: undefined } +1ms
cypress:server:util:socket_allowed allowing socket { localPort: 53734 } +44ms
cypress:server:server-base Got UPGRADE request from /ws +2ms
cypress:network:cors Parsed URL { port: '8080', tld: 'localhost', domain: '' } +3s
cypress:network:agent addRequest called { isHttps: false, href: 'http://localhost:8080/ws' } +3s
cypress:network:agent got family { family: 4, href: 'http://localhost:8080/ws' } +0ms
cypress:server:project received runnables { id: 'r1', title: '', root: true, type: 'suite', file: 'src/App_91.cy.jsx', retries: -1, _slow: 250, hooks: [], tests: [ { id: 'r2', title: '[App_91.cy.jsx]: renders learn react link', body: '() => {\n' + ' (0,_cypress_react__WEBPACK_IMPORTED_MODULE_1__.mount)( /*#__PURE__*/(0,react_jsx_dev_runtime__WEBPACK_IMPORTED_MODULE_2__.jsxDEV)(_App__WEBPACK_IMPORTED_MODULE_0__["default"], {}, void 0, false, {\n' + ' fileName: _jsxFileName,\n' + ' lineNumber: 5,\n' + ' columnNumber: 9\n' + ' }, undefined));\n' + ' cy.get("a").contains("React");\n' + '\n' + ' for (let i = 1; i < 10; i++) {\n' + ' cy.get("button").click();\n' + ' cy.get("p").contains(i);\n' + ' }\n' + '}', type: 'test', file: null, invocationDetails: [Object], currentRetry: 0, retries: -1, _slow: 250, hooks: [Array], _testConfig: [Object], _titlePath: [Array] } ], suites: [], runtimeConfig: {} } +2s
cypress:server:reporter trying to load reporter: spec +2s
cypress:server:reporter spec is Mocha reporter +0ms
cypress:server:project onMocha start +3msCypress 10 |
SummaryOk, I looked into it. It turns out creating a new tab (and associated processes) does impact the performance. Here's a branch with the 9.7.0 behavior, which is one tab -> run all specs in there, as opposed to the current implementation which is to create a new tab per spec.
There's a build of this branch you can install for linux x64, get it here. There's likely some bugs, but Note: these numbers are from 1 time runs - not the most scientific. To be more accurate, I'd run them all many times and take the average, but these are good enough to validate that opening a new tab is the primary reason we are slower. Toy repo
For the run time: (+18% Cypress 9.7.0, +37% Cypress 10.3.0) These numbers are a bit exaggerated, since I'm running 1 test per spec, which is not common. Normally, a single spec file will have 10s or 100s of tests. Either way, this experiment implies a ~300ms cost for creating a new tab, which is significant. picasso repo https://github.com/toptal/picasso (run locally)
It's 43% faster! I guess this mostly accounts for the 2x slowdown described. Cypress internal component tests (local, no parallel, no video)
Run locally. I turned off video, which makes everything slower in a linear fashion, so it shouldn't impact the relative differences. 30% is a solid improvement. DiscussionI think it's death by many a thousand paper cuts - just opening a new tab is not a single performance killer, but it looks like we do some cleanup between specs, and reset a bunch of things. I need to go deeper, but I suspect at least some of these things are absolutely necessary to guarantee spec isolation and determinism in E2E, which is why E2E has done the "new browser per spec" things since forever. I'll look more into this, and if it's necessary for component testing. As shown on my toy repo with 100 of the space spec, as as predicted, the actual test execution time is identical, since your production code and specs hasn't changed changed. Next Steps
Thanks for the patience and assistance here everyone! I think we are on the right track. I'll be focused on this until we have the performance issue solved. |
4 tasks
|
If anyone is following this and wants to give me some feedback, that'd be great. For the next 48 hours you can grab a pre-release for linux and mac here: 6e4d8cd (scroll down to the comments). It's possible E2E is a bit bugged in this build; I only tested it on component testing thoroughly, since the main goal is to fix the performance improvements. The change from one tab -> one tab per spec for component testing was to improve spec isolation, and make CT and E2E behave in a similar fashion. It actually resulted in a speed UP for E2E, which previously created a new browser for each spec. I think, ideally, we want the best of both worlds; faster component tests and better spec isolation, along with E2E consistency. I'll keep looking into this and post updates here. |
|
@lmiller1990 I have tested it out in Picasso and it works perfectly! 🚀 🎉 I really enjoyed following this issue and checking how you are handling it, especially since you share your thought process when debugging! Looking forward to this being released. Thank you for your work! Keep rocking! 💪 🙇 |
|
I tested 4 versions of Cypress with our smallest project that only has a handful of complex components that require many assertions to fully test, so most of them are fairly small test cases From my tests, it appears either There did not appear to be any significant reduction in test setup/cleanup duration from component tests being performed in one tab like with the Picasso and internal Cypress tests over consecutive runs of 10.3.0 and the Circle CI build (this macOS ARM build) on my machine with this small Vue project (both versions are already faster than Cypress 9 though!) TestsThese were run twice each with the times of the 2nd run recorded (so if anything is happening when "first time using All tests were with video recording disabled and using Cypress' Vue & Vite configuration Note: Skipped tests were disabled as they failed with Individual tests per component spec file
Total duration of each component spec file according to Cypress
Total test duration (Yarn's command execution timer)
|
|
@TomasSlama cool, thanks for the support and help testing!! @wopian can you confirm your table info a bit? When you say 10.3.0 you are referring to my build (not the live 10.3.0 on npm?) Those are some huge improvements, good to see! Just to clarify a bit more here - unfortunately it's not as simple as "just ship these changes". Opening a new tab clears a lot of things and better isolates the specs, which is a feature of any good test runner. The original goal of this change was to be more in line with E2E - you can image it'd be confusing if CT and E2E diverge. I will find out exactly what we do better each spec and find out how to do it in a faster way than what we do now. My next step is for me to dig deeper and isolate more closely what's really killing performance - I think it's more than just "creating a new tab", but also things like teardown/setup/rendering the runner UI. The end goal is to improve performance for both CT and E2E. The spec execution flow is at the core of Cypress. It needs a bit more investigation, but we are on the right track with this thread. We take this kind of regression (performance, and more importantly determinism and reliability) very seriously. Just to clarify, this is a performance regression for Component Testing - this change made E2E faster, since it use to spin up a new browser for each spec, now it just does a new tab. I am out next week, but after that I'm going to look into a more detailed technical breakdown, then we can actually start moving forward with some updates. The Component Testing "alpha" label was there for reason, and I'd like to move out of "beta" sooner than than later, but that means exercising great caution around mission critical things like performance and spec isolation. I will continue posting my updates here, I think it's fine to take a little time as long as we are clear with communication. This is still my main focus and will be until it's we are back to a comparable, if not faster, state than the 9.x Component Testing alpha. |
10.3.0 on npm One listed as Circle CI is the 10.3.0 from your build (used the ARM build linked in my comment). I can double check if it did correctly install and use your build or if I accidentally used an outdated build tomorrow |
|
Hi @lmiller1990! I just investigated a bit more and the root cause of the problem was that in our Since for the component testing we don't need the real routes (at least not most of the times) we have just changed it to use a mock router by default, e.g.: Thanks for your input! |
4 tasks
cypress-bot
bot
added
stage: needs review
|
Hello folks, an update - trying to get this through: #23104. It's awaiting internal review. It will likely have some edge cases on initial release - thus the experimental flag, but exactly what those limitations will be will clear when it lands. |
cypress-bot
bot
added
stage: blocked
and removed
stage: needs review
lmiller1990
changed the title
|
Another update - this is getting close, binaries available for testing: 293a0f4#comments To use it, you need to enable it. It's initially going out as an experimental feature. Example. export default defineConfig({
component: {
experimentalSingleTabRunMode: true // <= this is the flag to enable the single tab run mode
}
}) |
|
Ran the Darwin Arm64 build 3x on our two projects with and without the |
|
@lmiller1990 I have tested on Picasso and the results are from 112s 👉 62s. Great news 🎉. Thank you very much for your effort 🙇 |
|
Hi team, it's finally merged, it'll be in 10.6. Finally! Excited for this feature. |
|
Surprise off cycle release.... 10.6 is now out 🎉 |
|
@lmiller1990 kindly assist me here. thanks |
|
Hi @vision2divinity. I was out of office, sorry for the slow reply. I don't think your issue is related to this issue. It looks like the entire app is freezing? Are you experiencing this on a fresh project too, or just one particular project? I'd recommend filing a new, separate issue to get more eyes on this. |
|
I am running tests on cypress runner Please may I know what is reason behind this and |
|
@Prashika Please open an issue with a reproducible example if you'd like us to investigate on our end. This specific issue has been resolved. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Edit from @lmiller1990: this is getting worked on: #23104. Component Testing performance will be improved, initially released under an experimental flag to let use continue making incrementally performance updates. See latest comments for latest updates.
Edit: Try using the new
experimentalSingleTabRunModeflag in yourcomponentproperty incypress.config. It is up to 40% faster for larger suites. Read more in the docs here.Current behavior
After upgrading to the latest version, we have an issue with slowness when we run
cypress run --component. The tests themselves are not slow, but it seems there is some delay between specs runs as you can see in this video 👇 .cypress-issue.mp4
We were not able to detect the exact issue, but we would appreciate any help 🙏
I was trying to run the tests with
DEBUGand this might be potentially the issue 👇Desired behavior
The tests should run smoothly without any delay.
Test code to reproduce
We use cypress on public repo https://github.com/toptal/picasso.
We have cypress 10 on the current master.
for debugging it is faster to run
# these need to be run only once yarn yarn build:package yarn test:setup cypress run --componentCypress Version
10.1.0
Other
We use cypress with Happo for visual testing. The first idea was that it might be the problem since Happo didn't adapt to v10 yet, but it is not the case, I have tried to remove it from the repo and it didn't help.
The text was updated successfully, but these errors were encountered: