Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[system] Add browser benchmark #22923

Merged
merged 51 commits into from
Oct 12, 2020
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
c2d31cc
add emotion peer dependencies
mnajdova Sep 25, 2020
5ae933f
fixed types & tests
mnajdova Sep 25, 2020
18b0668
prettier
mnajdova Sep 25, 2020
f0ef95c
Merge branch 'next' of https://github.com/mui-org/material-ui into next
mnajdova Sep 28, 2020
c7bebb8
Merge branch 'next' of https://github.com/mui-org/material-ui into next
mnajdova Sep 28, 2020
92b2d6e
Merge branch 'next' of https://github.com/mui-org/material-ui into next
mnajdova Sep 28, 2020
13da531
Merge branch 'next' of https://github.com/mui-org/material-ui into next
mnajdova Sep 29, 2020
cf5d9a5
Merge branch 'next' of https://github.com/mui-org/material-ui into next
mnajdova Sep 29, 2020
b8d1291
Merge branch 'next' of https://github.com/mui-org/material-ui into next
mnajdova Sep 30, 2020
a9d8690
Merge branch 'next' of https://github.com/mui-org/material-ui into next
mnajdova Oct 2, 2020
497830a
Merge branch 'next' of https://github.com/mui-org/material-ui into next
mnajdova Oct 5, 2020
d50ea1e
Merge branch 'next' of https://github.com/mui-org/material-ui into next
mnajdova Oct 5, 2020
0f6a4cd
Merge branch 'next' of https://github.com/mui-org/material-ui into next
mnajdova Oct 7, 2020
616bd28
wip
mnajdova Oct 7, 2020
bcd581e
spaces
mnajdova Oct 7, 2020
0fde5b1
Added all system tests
mnajdova Oct 7, 2020
73105d0
moved wait-on and concurrently to devDependencies
mnajdova Oct 7, 2020
520cbc5
fixed & improvements
mnajdova Oct 7, 2020
b779a2a
moved serve + wait in perf
mnajdova Oct 7, 2020
13c3f6e
improved perf metrics
mnajdova Oct 7, 2020
4dff549
improved emotion example
mnajdova Oct 7, 2020
02fc972
added Profiler
mnajdova Oct 7, 2020
3ff75fd
Update test/perf/webpack.config.js
mnajdova Oct 7, 2020
95cad05
Olivier's feedback
mnajdova Oct 7, 2020
781c969
Merge branch 'feat/perf-browser-tests' of https://github.com/mnajdova…
mnajdova Oct 7, 2020
75425db
deduplicate
mnajdova Oct 7, 2020
37f61a9
removed fuu, prettier
mnajdova Oct 7, 2020
6abea87
lint
mnajdova Oct 7, 2020
e43ff18
lint errors
mnajdova Oct 8, 2020
500cfa2
renamed perf + tests to benchmark
mnajdova Oct 8, 2020
7127ed9
disable prefer default export from utils
mnajdova Oct 8, 2020
03a10e8
Fix mode confusion
eps1lon Oct 8, 2020
1446315
Exit script when it's done
eps1lon Oct 8, 2020
acda962
Remove unnecessary wrappers
eps1lon Oct 8, 2020
28f1ab0
Merge pull request #9 from eps1lon/feat/perf-browser-tests-review1
mnajdova Oct 8, 2020
b4c0e13
Olivier's comments
mnajdova Oct 11, 2020
32f1219
lint, added 1000 components per scenario
mnajdova Oct 11, 2020
f0c65a5
lint
mnajdova Oct 11, 2020
9763fbd
have sub millisecond measurments
oliviertassinari Oct 11, 2020
05e55ec
added noop and README
mnajdova Oct 11, 2020
60aa699
prettier
mnajdova Oct 12, 2020
0a835ef
Update benchmark/scripts/benchmark.js
mnajdova Oct 12, 2020
da8bf24
Merge pull request #10 from oliviertassinari/sub-ms
mnajdova Oct 12, 2020
73b06eb
Merge branch 'next' into feat/perf-browser-tests
mnajdova Oct 12, 2020
6a94b91
formatted median print
mnajdova Oct 12, 2020
91dcbfe
reverted dependencies
mnajdova Oct 12, 2020
ff5e451
Merge branch 'next' into feat/perf-browser-tests
mnajdova Oct 12, 2020
da39d72
conflicts
mnajdova Oct 12, 2020
4ed84c8
dependencies removed
mnajdova Oct 12, 2020
5292dba
Merge branch 'next' into feat/perf-browser-tests
mnajdova Oct 12, 2020
2df44b7
conflicts
mnajdova Oct 12, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
"jsonlint": "node scripts/jsonlint.js",
"lint": "eslint . --cache --report-unused-disable-directives --ext .js,.ts,.tsx",
"lint:ci": "eslint . --report-unused-disable-directives --ext .js,.ts,.tsx",
"perf": "yarn webpack --config test/perf/webpack.config.js --mode \"production\" && concurrently \"npx serve .\" \"npx wait-on http://localhost:5000/test/perf/debug && node scripts/perf.js \"",
eps1lon marked this conversation as resolved.
Show resolved Hide resolved
"prettier": "node ./scripts/prettier.js",
"prettier:all": "node ./scripts/prettier.js write",
"size:snapshot": "node scripts/sizeSnapshot/create",
Expand Down Expand Up @@ -189,5 +190,9 @@
"packages/*",
"docs",
"framer"
]
],
"dependencies": {
"concurrently": "^5.3.0",
"wait-on": "^5.2.0"
}
}
87 changes: 87 additions & 0 deletions scripts/perf.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
const puppeteer = require('puppeteer');
const { performance } = require('perf_hooks');

const SERVER = 'localhost';
const PORT = 5000;
const APP = 'test/perf/debug';

async function createBrowser() {
const browser = await puppeteer.launch();

return {
openPage: async url => {
const page = await browser.newPage();
await page.goto(url);

return {
close: async () => page.close(),
mnajdova marked this conversation as resolved.
Show resolved Hide resolved
};
},
close: async () => browser.close(),
};
}

async function runMeasures(
browser,
testCase,
times,
) {
const measures = [];

for (let i = 0; i < times; i++) {
var start = performance.now();
const page = await browser.openPage(`http://${SERVER}:${PORT}/${APP}?${testCase}`);
const end = performance.now();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way we can benchmark what's happening in the browser and send that back? Otherwise we'll add a lot of noise from what the browser is doing that is unrelated to the actual benchmark.

Seems like there's plenty of existing APIs for that: https://addyosmani.com/blog/puppeteer-recipes/#runtime-perf-metrics

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks much better, let me refactor to use the metrics()

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might also be interesting to compare these metrics() to React's Profiler component (see also How to use profiling in production).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactored using

    const perf = await page.evaluate(_ => {
      const { loadEventEnd, navigationStart } = performance.timing;
      return loadEventEnd - navigationStart;
    });

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me compare the numbers now with the React's Profiler component :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added the Profiler on the box-emotion example. Profiler result:

actualDuration: 19.274999969638884
baseDuration: 15.385000151582062
commitTime: 119.98000007588416
id: "box-emotion"
interactions: Set(0) {}
phase: "mount"
startTime: 98.64500002004206

perf result (10 different runs; this includes the component as well):

Box emotion:

69.00ms
68.00ms
90.00ms
78.00ms
44.00ms
44.00ms
57.00ms
56.00ms
56.00ms
56.00ms
-------------
Avg: 61.80ms

Let me try this on more examples to see if the relative differences are similar.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are more numbers:

  1. box-emotion
Box emotion:

69.00ms
67.00ms
77.00ms
93.00ms
43.00ms
57.00ms
56.00ms
56.00ms
57.00ms
43.00ms
-------------
Avg: 61.80ms
actualDuration: 18.609998980537057
baseDuration: 15.669998829253018
commitTime: 110.84500001743436
id: "box-emotion"
interactions: Set(0) {}
phase: "mount"
startTime: 90.1650000596419
  1. box-styled-components
Box styled-components:

67.00ms
63.00ms
64.00ms
80.00ms
97.00ms
61.00ms
61.00ms
47.00ms
48.00ms
48.00ms
-------------
Avg: 63.60ms
actualDuration: 15.089999069459736
baseDuration: 12.439999729394913
commitTime: 96.09999996609986
id: "naked-styled-components"
interactions: Set(0) {}
phase: "mount"
startTime: 79.29499994497746
  1. box @material-ui/styles
Box @material-ui/styles:

123.00ms
122.00ms
89.00ms
103.00ms
103.00ms
85.00ms
121.00ms
123.00ms
84.00ms
84.00ms
-------------
Avg: 103.70ms
actualDuration: 44.76000100839883
baseDuration: 41.65500064846128
commitTime: 130.44999993871897
id: "box-material-ui-system"
interactions: Set(0) {}
phase: "mount"
startTime: 83.49999994970858
  1. naked-styled-components
Box styled-components:

67.00ms
63.00ms
64.00ms
80.00ms
97.00ms
61.00ms
61.00ms
47.00ms
48.00ms
48.00ms
-------------
Avg: 63.60ms
actualDuration: 15.089999069459736
baseDuration: 12.439999729394913
commitTime: 96.09999996609986
id: "naked-styled-components"
interactions: Set(0) {}
phase: "mount"
startTime: 79.29499994497746

I am not sure how to compare the numbers of both as they are quite different... But I can conclude from both measurements for example that @material-ui/styles' Box is slower than the rest, not much else... We can try to render more instances of the components too, currently I am rendering 100 boxes. @eps1lon what do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd keep both: Browser metrics and React metrics.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added with 02fc972


measures.push(end - start);
await page.close();
}

return measures;
}

const printMeasures = (measures) => {
console.log('\nMeasures\n');

Object.keys(measures).forEach(measureKey => {
console.log(`\n${measureKey}:\n`);

let sum = 0;
const totalNum = measures[measureKey].length;

measures[measureKey].forEach(measure => {
sum += measure;
console.log(`${measure.toFixed(2)}ms`);
});

console.log("-------------");
console.log(`Avg: ${Number(sum/totalNum).toFixed(2)}ms\n`)
});
}

async function run(argv) {
const browser = await createBrowser();
let measures = {};

try {
measures['@material-ui/system colors'] = await runMeasures(browser, './material-ui-system-colors/index.js', 10);
measures['styled-system colors'] = await runMeasures(browser, './styled-system-colors/index.js', 10);
measures['@material-ui/system spaces'] = await runMeasures(browser, './material-ui-system-spaces/index.js', 10);
measures['styled-system spaces'] = await runMeasures(browser, './styled-system-spaces/index.js', 10);
measures['@material-ui/system compose'] = await runMeasures(browser, './material-ui-system-compose/index.js', 10);
measures['styled-system compose'] = await runMeasures(browser, './styled-system-compose/index.js', 10);
measures['@material-ui/core all-inclusive'] = await runMeasures(browser, './material-ui-system-all-inclusive/index.js', 10)
measures['styled-components Box + @material-ui/system'] = await runMeasures(browser, './styled-components-box-material-ui-system/index.js', 10);
measures['styled-components Box + styled-system'] = await runMeasures(browser, './styled-components-box-styled-system', 10);
measures['Box emotion'] = await runMeasures(browser, './box-emotion/index.js', 10);
measures['Box @material-ui/styles'] = await runMeasures(browser, './box-material-ui-styles/index.js', 10);
measures['Box styled-components'] = await runMeasures(browser, './box-styled-components/index.js', 10);
measures['Naked styled-components'] = await runMeasures(browser, './naked-styled-components/index.js', 10);
} finally {
await browser.close();
}

printMeasures(measures);
}

run();
13 changes: 13 additions & 0 deletions test/perf/debug.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<!DOCTYPE html>
<html>
<head>
<title>Perf scenario</title>
<meta charset="utf-8" />
<meta name="viewport" content="initial-scale=1, width=device-width" />
<style>body { background-color: white; }</style>
</head>
<body>
<div id="root"></div>
<script src="../../tmp/tests.js"></script>
</body>
</html>
19 changes: 19 additions & 0 deletions test/perf/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import React from 'react';
import ReactDOM from 'react-dom';

// Get all the tests specifically written for preventing regressions.
const requirePerfTests = require.context('./tests', true, /(js|ts|tsx)$/);

const rootEl = document.getElementById('root');

// ./Button/index.js
// ./Dialog/index.js

const testSuitePath = window.location.search.replace("?", "");

const Component = requirePerfTests(testSuitePath).default;

ReactDOM.render(
<Component />,
rootEl,
);
26 changes: 26 additions & 0 deletions test/perf/tests/box-emotion/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@

mnajdova marked this conversation as resolved.
Show resolved Hide resolved
import React from 'react';
import { ThemeProvider as EmotionTheme } from 'emotion-theming';
import { styleFunction } from '@material-ui/core/Box';

const materialSystemTheme = createMuiTheme();
const BoxEmotion = styledEmotion('div')(styleFunction);

const App = () => {
return (
<EmotionTheme theme={materialSystemTheme}>
<BoxEmotion
color="primary.main"
bgcolor="background.paper"
fontFamily="h6.fontFamily"
fontSize={['h6.fontSize', 'h4.fontSize', 'h3.fontSize']}
p={[2, 3, 4]}
fuu={Math.round(Math.random() * 10000)}
mnajdova marked this conversation as resolved.
Show resolved Hide resolved
>
emotion
</BoxEmotion>
</EmotionTheme>
);
}

export default App;
25 changes: 25 additions & 0 deletions test/perf/tests/box-material-ui-styles/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@

import React from 'react';
import { ThemeProvider as StylesThemeProvider } from '@material-ui/styles';
import BoxStyles from '@material-ui/core/Box';

const materialSystemTheme = createMuiTheme();

const App = () => {
return (
<StylesThemeProvider theme={materialSystemTheme}>
<BoxStyles
color="primary.main"
bgcolor="background.paper"
fontFamily="h6.fontFamily"
fontSize={['h6.fontSize', 'h4.fontSize', 'h3.fontSize']}
p={[2, 3, 4]}
fuu={Math.round(Math.random() * 10000)}
>
@material-ui/styles
</BoxStyles>
</StylesThemeProvider>
);
}

export default App;
28 changes: 28 additions & 0 deletions test/perf/tests/box-styled-components/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
import React from 'react';
import { createMuiTheme } from '@material-ui/core/styles';
import { styleFunction } from '@material-ui/core/Box';
import styledComponents, {
ThemeProvider as StyledComponentsThemeProvider,
} from 'styled-components';

const materialSystemTheme = createMuiTheme();
const BoxStyleComponents = styledComponents('div')(styleFunction);

const App = () => {
return (
<StyledComponentsThemeProvider theme={materialSystemTheme}>
<BoxStyleComponents
color="primary.main"
bgcolor="background.paper"
fontFamily="h6.fontFamily"
fontSize={['h6.fontSize', 'h4.fontSize', 'h3.fontSize']}
p={[2, 3, 4]}
fuu={Math.round(Math.random() * 10000)}
>
styled-components
</BoxStyleComponents>
</StyledComponentsThemeProvider>
);
}

export default App;
18 changes: 18 additions & 0 deletions test/perf/tests/material-ui-system-all-inclusive/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@

import React from 'react';
import { styleFunction } from '@material-ui/core/Box';

const App = () => {
const result = styleFunction({
theme: materialSystemTheme,
color: 'primary.main',
bgcolor: 'background.paper',
fontFamily: 'h6.fontFamily',
fontSize: ['h6.fontSize', 'h4.fontSize', 'h3.fontSize'],
p: [2, 3, 4],
});

return null;
}

export default App;
13 changes: 13 additions & 0 deletions test/perf/tests/material-ui-system-colors/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import React from 'react';
import { palette } from '@material-ui/system';

const App = () => {
const result = palette({
theme: {},
bgcolor: ['red', 'blue'],
});

return null;
}

export default App;
19 changes: 19 additions & 0 deletions test/perf/tests/material-ui-system-compose/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import React from 'react';
import { spacing, palette, typography, compose } from '@material-ui/system';

const materialSystem = compose(palette, spacing, typography);

const App = () => {
const result = materialSystem({
theme: materialSystemTheme,
color: 'primary.main',
bgcolor: 'background.paper',
fontFamily: 'h6.fontFamily',
fontSize: ['h6.fontSize', 'h4.fontSize', 'h3.fontSize'],
p: [2, 3, 4],
});

return null;
}

export default App;
13 changes: 13 additions & 0 deletions test/perf/tests/material-ui-system-spaces/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import React from 'react';
import { spacing } from '@material-ui/system';

const App = () => {
const result = spacing({
theme: {},
p: [1, 2, 3],
});

return null;
}

export default App;
28 changes: 28 additions & 0 deletions test/perf/tests/naked-styled-components/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
import React from 'react';
import { createMuiTheme } from '@material-ui/core/styles';
import { spacing } from '@material-ui/system';
import styledComponents, {
ThemeProvider as StyledComponentsThemeProvider,
} from 'styled-components';

const materialSystemTheme = createMuiTheme();
const NakedStyleComponents = styledComponents('div')(spacing);
eps1lon marked this conversation as resolved.
Show resolved Hide resolved

const App = () => {
return (
<StyledComponentsThemeProvider theme={materialSystemTheme}>
<NakedStyleComponents
color="primary.main"
bgcolor="background.paper"
fontFamily="h6.fontFamily"
fontSize={['h6.fontSize', 'h4.fontSize', 'h3.fontSize']}
p={[2, 3, 4]}
fuu={Math.round(Math.random() * 10000)}
>
styled-components
</NakedStyleComponents>
</StyledComponentsThemeProvider>
);
}

export default App;
29 changes: 29 additions & 0 deletions test/perf/tests/styled-components-box-material-ui-system/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import React from 'react';
import { createMuiTheme } from '@material-ui/core/styles';
import styledComponents, {
ThemeProvider as StyledComponentsThemeProvider,
} from 'styled-components';
import { spacing, palette, typography, compose } from '@material-ui/system';

const materialSystem = compose(palette, spacing, typography);
const materialSystemTheme = createMuiTheme();
const BoxMaterialSystem = styledComponents('div')(materialSystem);

const App = () => {
return (
<StyledComponentsThemeProvider theme={materialSystemTheme}>
<BoxMaterialSystem
color="primary.main"
bgcolor="background.paper"
fontFamily="h6.fontFamily"
fontSize={['h6.fontSize', 'h4.fontSize', 'h3.fontSize']}
p={[2, 3, 4]}
>
@material-ui/system
</BoxMaterialSystem>
</StyledComponentsThemeProvider>
);
}

export default App;

33 changes: 33 additions & 0 deletions test/perf/tests/styled-components-box-styled-system/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
import React from 'react';
import styledComponents, {
ThemeProvider as StyledComponentsThemeProvider,
} from 'styled-components';
import { space, color, fontFamily, fontSize, compose } from 'styled-system';

const styledSystem = compose(color, space, fontFamily, fontSize);
const BoxStyledSystem = styledComponents('div')(styledSystem);

const styledSystemTheme = createMuiTheme();
styledSystemTheme.breakpoints = ['40em', '52em', '64em'];
styledSystemTheme.colors = styledSystemTheme.palette;
styledSystemTheme.fontSizes = styledSystemTheme.typography;
styledSystemTheme.fonts = styledSystemTheme.typography;

const App = () => {
return (
<StyledComponentsThemeProvider theme={styledSystemTheme}>
<BoxStyledSystem
color="primary.main"
bg="background.paper"
fontFamily="h6.fontFamily"
fontSize={['h6.fontSize', 'h4.fontSize', 'h3.fontSize']}
p={[2, 3, 4]}
>
styled-system
</BoxStyledSystem>
</StyledComponentsThemeProvider>
);
}

export default App;

13 changes: 13 additions & 0 deletions test/perf/tests/styled-system-colors/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import React from 'react';
import { color } from 'styled-system';

const App = () => {
const result = color({
theme: {},
bg: ['red', 'blue'],
});;

return null;
}

export default App;
Loading