Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Add benchmark script #2683

Merged
merged 3 commits into from
Sep 29, 2021
Merged

[core] Add benchmark script #2683

merged 3 commits into from
Sep 29, 2021

Conversation

m4theushw
Copy link
Member

@m4theushw m4theushw commented Sep 24, 2021

I added this script to compare the performance of the old virtualization with the new one being developed in #2673

Related to #2175 and #1933

Running yarn benchmark:browser gave the following metrics:

React Virtualized:
  Vertical scroll:
       Min: 24.257 fps (σ = 2.077)
       Max: 56.042 fps (σ = 2.943)
    Median: 47.334 fps (σ = 0.207)
      Mean: 46.448 fps (σ = 0.735)
  Horizontal scroll:
       Min: 21.791 fps (σ = 2.694)
       Max: 50.951 fps (σ = 4.161)
    Median: 43.401 fps (σ = 0.258)
      Mean: 41.983 fps (σ = 0.699)
AG Grid:
  Vertical scroll:
       Min: 20.578 fps (σ = 2.321)
       Max: 56.132 fps (σ = 0.439)
    Median: 49.325 fps (σ = 1.841)
      Mean: 46.326 fps (σ = 1.955)
  Horizontal scroll:
       Min: 13.538 fps (σ = 2.799)
       Max: 57.005 fps (σ = 0.999)
    Median: 22.289 fps (σ = 0.481)
      Mean: 23.489 fps (σ = 0.578)
MUI+:
  Vertical scroll:
       Min: 31.212 fps (σ = 5.834)
       Max: 55.935 fps (σ = 1.47)
    Median: 48.834 fps (σ = 0.239)
      Mean: 48.094 fps (σ = 0.468)
  Horizontal scroll:
       Min: 31.898 fps (σ = 4.358)
       Max: 57.549 fps (σ = 0.519)
    Median: 46.95 fps (σ = 0.286)
      Mean: 46.52 fps (σ = 0.206)
DataGridPro:
  Vertical scroll:
       Min: 11.785 fps (σ = 1.545)
       Max: 57.056 fps (σ = 1.207)
    Median: 24.246 fps (σ = 0.361)
      Mean: 27.491 fps (σ = 0.256)
  Horizontal scroll:
       Min: 7.904 fps (σ = 0.951)
       Max: 56.24 fps (σ = 1.031)
    Median: 12.53 fps (σ = 0.368)
      Mean: 13.415 fps (σ = 0.099)

Resolution tested: 1280x720

@m4theushw m4theushw added the core Infrastructure work going on behind the scenes label Sep 24, 2021
@flaviendelangle
Copy link
Member

@DanailH
Copy link
Member

DanailH commented Sep 29, 2021

Should we store or persist the previous values somehow, or just have a set threshold so we know if we decrease the performance (it can be as a follow-up)?

@m4theushw
Copy link
Member Author

Should we store or persist the previous values somehow, or just have a set threshold so we know if we decrease the performance (it can be as a follow-up)?

@DanailH I guess that to do that we will need to run inside mocha to be able to do assertions. But yeah, having a better way to check performance degradation would be nice.


Is it based on https://github.com/DanailH/mui-datagrid-measurements ?

@flaviendelangle It was based on https://github.com/Janpot/mui-plus/blob/master/scripts/benchmark.ts. The mentioned repo uses the WebVitals API which measure other things, not the FPS.

@m4theushw m4theushw merged commit 71cc0d3 into mui:next Sep 29, 2021
@m4theushw m4theushw deleted the benchmark branch September 29, 2021 14:46
@oliviertassinari
Copy link
Member

oliviertassinari commented Nov 22, 2021

@m4theushw it's awesome that have this in :)

Should we store or persist the previous values somehow

In the main repository, we have a markdown where we console log the output. This gives a beginning of a comparison point.


https://github.com/mui-org/material-ui-x/blob/1d3a44ea00a7f753ba13af92951b758fb695b47e/benchmark/browser/scripts/benchmark.js#L120-L122

Haha @Janpot thanks for the help on this one.


@m4theushw I'm not sure how to measure it, but reading the part about the virtualization improvements in mui/material-ui#29590, I believe we miss an important metric (maybe the most important one). From how I understand it, FPS tells us about how much the main thread of the UI is blocked. However, it doesn't tell us how fast the rows are displayed. With React 18 interruptable rendering, I suspect that increasingly we will go to 60 FPS but this won't mean that the virtualization feels more performant, not if it still takes 200ms for a row that is offscreen to get mounted. I recall that Damien was putting minimizing the white zone as its ultimate target, even if it lead to lower FPS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Infrastructure work going on behind the scenes performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants