fix: test runner reporter performance

[tgriesser]

• Closes Tests slow down over time when performing several XHR requests calls in single test #6783

User facing changelog

Improved performance with larger numbers of tests.

Additional details

Did some digging into rendering bottlenecks in the reporter, found a number of things needing addressing, the lowest hanging fruit are the lazy initial rendering of runnables, and the custom styling of the reporter scrollbar. The PR removes the custom styling for the scrollbar & renders all runnables from the start.

Uses this test case, updating the COUNT as seen in the table screenshot below:

const COUNT = 100

describe('renders tests', () => {
  Cypress._.times(COUNT, (n) => {
    it(`${n} test`, () => {
      expect(true).to.be.true
    })
  })
})

Recorded benchmarks for removing the progressive rendering of the runnable list, as well as the style fixes - and then both combined. My hunch here is that the reason we needed to add the incremental node rendering has been addressed by more recent React versions.

Also manually confirmed that it fixes the experience in: #6783

How has the user experience changed?

PR Tasks

• Have tests been added/updated?
• Has the original issue or this PR been tagged with a release in ZenHub?

[cypress-bot]

Thanks for taking the time to open a PR!

• Create a Draft Pull Request if your PR is not ready for review. Mark the PR as Ready for Review when you're ready for a Cypress team member to review the PR.
• Become familiar with the Code Review Checklist for guidelines on coding standards and what needs to be done before a PR can be merged.

[cypress]

---

Test summary

450 • 0 • 6 • 0 • 0

---

Run details

Project	cypress
Status	Passed
Commit	3a68ac9
Started	Jul 28, 2021 3:57 PM
Ended	Jul 28, 2021 4:06 PM
Duration	09:08 💡
OS	Linux Debian - 10.8
Browser	Electron 89

View run in Cypress Dashboard ➡️

---

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

[jennifer-shehane]

@tgriesser

1. I can see that this improves performance for Tests slow down over time when performing several XHR requests calls in single test #6783 but doesn't completely account for all slowdown. The test still runs progressively slower towards the end of the test than it did at the beginning, although the original example is a bit extreme. The type command is about ~400ms slower towards the end compared to the first type command instead of ~3000ms slower (before this change)! So this is a big improvement.
2. I don't feel super great about not displaying a scrollbar for scrollable content. This is poor usability. Anything we can do about this?
3. The performance seems to be slightly worse in Firefox after this change? 😬
4. It would be 💯 if we had a way to test this. I know this is the hardest part though. There's kind of nothing stopping someone in the future from adding some slight change and just blowing out the performance again.

I'd like to explore points 2 and 3, but overall I still think this may be an improvement to merge in even with those downsides. Would love someone else on the team to review as well.

Chromium browsers

Before

Finished in 223 seconds

Using cypress-timings to show the type command duration of first type versus last type in test.

After

Finished in 107 seconds

Using cypress-timings to show the type command duration of first type versus last type in test.

Firefox browser

Before

Finished in 128 seconds

Using cypress-timings to show the type command duration of first type versus last type in test.

After

Finished in 151 seconds

Using cypress-timings to show the type command duration of first type versus last type in test.

[tgriesser]

@jennifer-shehane Thanks for the detailed review!

I don't feel super great about not displaying a scrollbar for scrollable content. This is poor usability. Anything we can do about this?

We could display the normal scrollbar unstyled (screenshot), or enable the scrollbar if you're not auto-scrolling?

I didn't know if we wanted to keep the current styling to keep it consistent with how it is now, so I opted to disable it entirely while the specs are running and show it when you're done and are more likely to find it useful. If we virtualize (see point below), we'll probably want to disable it entirely as well.

The performance seems to be slightly worse in Firefox after this change? 😬

Interesting about Firefox, will need to take a look and see if it's the CSS or the runnable renderings that causes this regression

The test still runs progressively slower towards the end of the test than it did at the beginning, although the original example is a bit extreme.

Yes, didn't anticipate this would be the only solution - this was just the simplest first pass, also looking into another approach of virtualizing the runner content while it's executing, which I believe should have a pretty massive improvement - but the implementation remains to be seen.

It would be 💯 if we had a way to test this. I know this is the hardest part though. There's kind of nothing stopping someone in the future from adding some slight change and just blowing out the performance again.

We could probably create a bounds of what we roughly expect to be correct timings and run this in its own isolated e2e spec. Might be flaky but I think should be doable 🤔

[jennifer-shehane]

@tgriesser I feel like the original intent for styling the scrollbar years ago was to make it slimmer than the native one and standardize across OS. I think it'd probably be ok to just remove the styling altogether. It doesn't even work in Firefox.

[jennifer-shehane]

@tgriesser Is this ready for rereview? I'm a little unclear.

[tgriesser]

Ran in Firefox w/ your test cases & prod build of react yarn build-prod in runner package, seeing about equivalent performance:

Before	After

Also was having trouble figuring out how to get this tested without being super flaky, but I think as long as we just make it a rule to not add the custom webkit browser scrollbar styling we should be ok.

[jennifer-shehane]

Um, yah this is a lot faster with the build-prod than I even realized. 💯 💯 💯 💯 Thanks @tgriesser