Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmark: pre-generate data set for URL benchmarks and use WPT test data #24302

Closed
wants to merge 3 commits into from

Conversation

joyeecheung
Copy link
Member

benchmark: pre-generate data set for URL benchmarks

This patch:

  • Introduces common.bakeUrlData which can be used to pre-generate
    the data set for the URL benchmarks to loop through instead of
    looping over a constant.
  • Add the option to use WPT data in benchmarks for better diversity
    in the input
  • Add the option to benchmark URL parsing with base URLs (whatwg only)
  • Moves the data in benchmark/fixtures/url-inputs.js to
    benchmark/common.js

benchmark: support URL inputs in create-clientrequest

This patch adds the option in the create-clientrequest benchmark
to accept URL inputs (as strings or as URL objects) so we can
measure the impact of URL parsing in a more sophisticated use case.

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • commit message follows commit guidelines

This patch:

- Introduces `common.bakeUrlData` which can be used to pre-generate
  the data set for the URL benchmarks to loop through instead of
  looping over a constant.
- Add the option to use WPT data in benchmarks for better diversity
  in the input
- Add the option to benchmark URL parsing with base URLs (whatwg only)
- Moves the data in `benchmark/fixtures/url-inputs.js` to
  `benchmark/common.js`
This patch adds the option in the create-clientrequest benchmark
to accept URL inputs (as strings or as URL objects) so we can
measure the impact of URL parsing in a more sophisticated use case.
@joyeecheung joyeecheung added the whatwg-url Issues and PRs related to the WHATWG URL implementation. label Nov 11, 2018
@nodejs-github-bot nodejs-github-bot added benchmark Issues and PRs related to the benchmark subsystem. http Issues or PRs related to the http subsystem. querystring Issues and PRs related to the built-in querystring module. test Issues and PRs related to the tests. url Issues and PRs related to the legacy built-in url module. labels Nov 11, 2018
@joyeecheung
Copy link
Member Author

const bench = common.createBenchmark(main, {
len: [1, 8, 16, 32, 64, 128],
n: [1e6]
Copy link
Contributor

@mscdex mscdex Nov 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is n being removed in general in this PR? If we're not going to be looping over the same input for each input, then we should at least loop n times over the entire set of inputs.

Copy link
Member Author

@joyeecheung joyeecheung Nov 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mscdex That should already been covered by the combination e and --runs (of the compare.js): the size of the input is dependent on e so to get a size of 1e6 we can just set e to roughly Math.log(1e6 / 200, 2) (all the data are pre-generated, so when e is 1 and type is long it will iterate over the same kind of string 400 times. Only the type wpt gives you a nonhomogenous array), but I don't think we actually need that many iterations for a single benchmark run, the n was set when Crankshaft existed, and currently I don't observe a significance in the value of the n once it's large enough (by default it's roughly 800/400 with/without base). Instead we can just use --runs to get more samples, but the default 30 is already a pretty good sample size (at least it's the magic number taught in statistics textbooks). So with the default setting for each input you run through the same code roughly 30 * 400 times with the same binary

@joyeecheung
Copy link
Member Author

joyeecheung commented Nov 11, 2018

@mscdex BTW this is the benchmark results I got for #24218 with this benchmark after setting e to 1, 4, 8 (so the size of the array in every single run is 200 * 2, 200 * 2 ** 4, 200 * 2 ** 8). As shown in the accuracy (the rightmost three columns) below the size of the array in one iteration doesn't matter much at least when it's bigger than 200 * 2 and when there are 30 sample runs for each configuration - the difference of accuracy is only roughly ±1% when all of the risks are just randomly small. (I also ran it through legacy which was not touched by #24218 at all and as expected it doesn't show any false positives)

See results
                                                                                          confidence improvement accuracy (*)   (**)  (***)
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='auth' withBase='false'                      -1.70 %       ±2.31% ±3.09% ±4.07%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='auth' withBase='true'                        0.98 %       ±2.00% ±2.67% ±3.49%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='dot' withBase='false'                        1.20 %       ±2.61% ±3.49% ±4.57%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='dot' withBase='true'                         0.71 %       ±1.13% ±1.50% ±1.95%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='file' withBase='false'                      -0.05 %       ±2.49% ±3.31% ±4.31%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='file' withBase='true'                       -1.76 %       ±3.42% ±4.58% ±6.00%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='idn' withBase='false'                       -0.32 %       ±1.17% ±1.56% ±2.03%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='idn' withBase='true'                        -0.06 %       ±4.55% ±6.05% ±7.88%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='javascript' withBase='false'                 1.84 %       ±2.98% ±3.97% ±5.18%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='javascript' withBase='true'                  0.39 %       ±1.32% ±1.76% ±2.30%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='long' withBase='false'                      -0.95 %       ±1.79% ±2.39% ±3.12%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='long' withBase='true'                        1.55 %       ±2.30% ±3.07% ±4.02%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='percent' withBase='false'                    1.60 %       ±1.83% ±2.44% ±3.18%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='percent' withBase='true'                     1.03 %       ±2.66% ±3.55% ±4.64%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='short' withBase='false'                      0.34 %       ±1.15% ±1.53% ±2.00%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='short' withBase='true'                       1.11 %       ±2.74% ±3.64% ±4.74%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='wpt' withBase='false'                        3.88 %       ±4.19% ±5.63% ±7.43%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='wpt' withBase='true'                         0.24 %       ±1.52% ±2.03% ±2.65%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='ws' withBase='false'                        -1.93 %       ±3.04% ±4.06% ±5.33%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=1 type='ws' withBase='true'                          1.22 %       ±2.92% ±3.89% ±5.08%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='auth' withBase='false'                      -0.94 %       ±4.24% ±5.64% ±7.34%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='auth' withBase='true'                        2.38 %       ±4.23% ±5.63% ±7.34%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='dot' withBase='false'                        0.46 %       ±3.97% ±5.28% ±6.88%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='dot' withBase='true'                         1.16 %       ±2.91% ±3.90% ±5.13%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='file' withBase='false'                       0.00 %       ±2.32% ±3.08% ±4.01%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='file' withBase='true'                        1.14 %       ±3.12% ±4.15% ±5.42%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='idn' withBase='false'                        0.94 %       ±2.70% ±3.59% ±4.67%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='idn' withBase='true'                        -0.89 %       ±2.23% ±2.97% ±3.88%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='javascript' withBase='false'                 0.60 %       ±2.20% ±2.93% ±3.84%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='javascript' withBase='true'                 -0.61 %       ±1.59% ±2.12% ±2.77%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='long' withBase='false'                      -0.11 %       ±2.38% ±3.17% ±4.12%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='long' withBase='true'                        0.34 %       ±1.90% ±2.53% ±3.29%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='percent' withBase='false'                   -0.98 %       ±1.61% ±2.14% ±2.79%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='percent' withBase='true'                    -1.91 %       ±2.26% ±3.03% ±3.98%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='short' withBase='false'                      0.08 %       ±1.60% ±2.13% ±2.77%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='short' withBase='true'                       1.04 %       ±2.15% ±2.87% ±3.75%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='wpt' withBase='false'                        0.34 %       ±2.47% ±3.30% ±4.30%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='wpt' withBase='true'                        -2.43 %       ±3.25% ±4.35% ±5.74%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='ws' withBase='false'                         0.18 %       ±4.31% ±5.74% ±7.47%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=4 type='ws' withBase='true'                         -1.58 %       ±4.90% ±6.53% ±8.50%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='auth' withBase='false'                      -0.54 %       ±2.15% ±2.86% ±3.72%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='auth' withBase='true'                       -0.23 %       ±2.24% ±2.97% ±3.87%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='dot' withBase='false'                       -1.07 %       ±2.72% ±3.62% ±4.71%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='dot' withBase='true'                         0.06 %       ±1.87% ±2.48% ±3.23%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='file' withBase='false'                      -1.98 %       ±2.31% ±3.07% ±4.01%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='file' withBase='true'                       -1.23 %       ±1.80% ±2.39% ±3.11%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='idn' withBase='false'                        0.57 %       ±2.26% ±3.01% ±3.92%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='idn' withBase='true'                         0.42 %       ±2.52% ±3.35% ±4.36%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='javascript' withBase='false'                 0.60 %       ±1.77% ±2.35% ±3.07%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='javascript' withBase='true'                  0.68 %       ±2.38% ±3.17% ±4.13%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='long' withBase='false'                      -0.04 %       ±1.94% ±2.58% ±3.37%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='long' withBase='true'                       -0.40 %       ±2.14% ±2.85% ±3.71%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='percent' withBase='false'                   -1.13 %       ±2.54% ±3.37% ±4.39%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='percent' withBase='true'                     0.87 %       ±2.37% ±3.15% ±4.11%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='short' withBase='false'                      0.76 %       ±2.62% ±3.49% ±4.54%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='short' withBase='true'                       0.48 %       ±2.00% ±2.66% ±3.46%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='wpt' withBase='false'                        1.32 %       ±2.14% ±2.85% ±3.73%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='wpt' withBase='true'                        -1.32 %       ±2.83% ±3.77% ±4.93%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='ws' withBase='false'                         0.38 %       ±2.40% ±3.20% ±4.16%
 url/legacy-vs-whatwg-url-parse.js method='legacy' e=8 type='ws' withBase='true'                          0.03 %       ±1.64% ±2.18% ±2.83%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='false'              ***    -11.25 %       ±1.80% ±2.40% ±3.13%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='true'               ***    -14.26 %       ±3.53% ±4.70% ±6.12%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='false'               ***    -14.39 %       ±3.99% ±5.32% ±6.96%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='true'                ***    -14.00 %       ±2.17% ±2.90% ±3.79%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='false'              ***    -14.94 %       ±2.36% ±3.14% ±4.08%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='true'               ***    -12.21 %       ±1.77% ±2.35% ±3.07%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='false'               ***    -15.15 %       ±1.80% ±2.39% ±3.12%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='true'                ***    -12.46 %       ±3.01% ±4.00% ±5.22%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='false'        ***    -15.95 %       ±1.90% ±2.53% ±3.29%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='true'         ***    -15.11 %       ±2.69% ±3.60% ±4.71%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='false'              ***     -8.21 %       ±1.39% ±1.85% ±2.41%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='true'               ***     -9.36 %       ±1.73% ±2.30% ±3.01%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='false'           ***    -16.61 %       ±2.53% ±3.36% ±4.38%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='true'            ***    -12.59 %       ±3.49% ±4.66% ±6.10%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='false'             ***    -15.16 %       ±2.32% ±3.09% ±4.03%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='true'              ***    -14.91 %       ±2.94% ±3.92% ±5.14%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='false'               ***    -14.07 %       ±2.84% ±3.79% ±4.95%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='true'                ***     -9.35 %       ±1.54% ±2.05% ±2.68%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='false'                ***    -15.84 %       ±1.35% ±1.80% ±2.35%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='true'                 ***    -13.88 %       ±2.04% ±2.72% ±3.54%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='auth' withBase='false'              ***    -11.49 %       ±2.11% ±2.82% ±3.68%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='auth' withBase='true'               ***     -9.76 %       ±1.93% ±2.57% ±3.35%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='dot' withBase='false'               ***    -15.30 %       ±2.26% ±3.01% ±3.92%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='dot' withBase='true'                ***    -14.11 %       ±2.67% ±3.56% ±4.64%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='file' withBase='false'              ***     -8.92 %       ±4.15% ±5.58% ±7.35%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='file' withBase='true'               ***    -13.43 %       ±2.06% ±2.74% ±3.58%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='idn' withBase='false'               ***    -12.58 %       ±1.56% ±2.08% ±2.71%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='idn' withBase='true'                ***    -15.71 %       ±2.69% ±3.61% ±4.75%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='javascript' withBase='false'        ***    -14.87 %       ±1.90% ±2.52% ±3.29%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='javascript' withBase='true'         ***    -12.78 %       ±2.53% ±3.38% ±4.42%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='long' withBase='false'              ***     -6.34 %       ±2.28% ±3.03% ±3.94%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='long' withBase='true'               ***     -9.13 %       ±2.87% ±3.81% ±4.96%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='percent' withBase='false'           ***    -14.49 %       ±1.68% ±2.25% ±2.94%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='percent' withBase='true'            ***    -13.99 %       ±2.89% ±3.85% ±5.02%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='short' withBase='false'             ***    -13.38 %       ±1.94% ±2.58% ±3.37%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='short' withBase='true'              ***    -11.86 %       ±2.47% ±3.29% ±4.29%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='wpt' withBase='false'               ***    -13.37 %       ±2.56% ±3.40% ±4.43%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='wpt' withBase='true'                ***    -12.79 %       ±2.33% ±3.12% ±4.08%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='ws' withBase='false'                ***    -12.10 %       ±2.01% ±2.68% ±3.49%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=4 type='ws' withBase='true'                 ***    -12.33 %       ±1.30% ±1.72% ±2.24%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='auth' withBase='false'              ***    -12.70 %       ±2.21% ±2.94% ±3.84%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='auth' withBase='true'               ***    -12.98 %       ±1.93% ±2.57% ±3.35%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='dot' withBase='false'               ***    -14.46 %       ±2.31% ±3.08% ±4.02%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='dot' withBase='true'                ***    -14.12 %       ±1.88% ±2.50% ±3.26%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='file' withBase='false'              ***    -12.76 %       ±1.95% ±2.60% ±3.39%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='file' withBase='true'               ***    -12.59 %       ±1.94% ±2.60% ±3.41%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='idn' withBase='false'               ***    -12.28 %       ±2.47% ±3.29% ±4.29%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='idn' withBase='true'                ***    -14.26 %       ±2.60% ±3.46% ±4.50%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='javascript' withBase='false'        ***    -18.45 %       ±2.80% ±3.73% ±4.86%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='javascript' withBase='true'         ***    -14.16 %       ±2.20% ±2.93% ±3.81%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='long' withBase='false'              ***     -6.78 %       ±1.84% ±2.46% ±3.21%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='long' withBase='true'               ***     -6.99 %       ±1.88% ±2.50% ±3.25%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='percent' withBase='false'           ***    -15.37 %       ±2.10% ±2.80% ±3.64%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='percent' withBase='true'            ***    -13.10 %       ±2.28% ±3.04% ±3.98%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='short' withBase='false'             ***    -14.13 %       ±1.73% ±2.30% ±3.00%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='short' withBase='true'              ***    -12.76 %       ±2.15% ±2.86% ±3.73%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='wpt' withBase='false'               ***    -14.31 %       ±2.08% ±2.77% ±3.61%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='wpt' withBase='true'                ***    -12.25 %       ±1.88% ±2.50% ±3.25%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='ws' withBase='false'                ***    -14.86 %       ±2.04% ±2.71% ±3.53%
 url/legacy-vs-whatwg-url-parse.js method='whatwg' e=8 type='ws' withBase='true'                 ***    -12.15 %       ±2.37% ±3.15% ±4.10%

For comparison, with the old benchmark the accuracy isn't really any different from when e=1 above even though it runs much longer

See results
                                                                              confidence improvement accuracy (*)   (**)  (***)
 url/legacy-vs-whatwg-url-parse.js n=100000 method='whatwg' type='auth'              ***    -15.23 %       ±2.23% ±2.97% ±3.87%
 url/legacy-vs-whatwg-url-parse.js n=100000 method='whatwg' type='dot'               ***    -14.43 %       ±2.85% ±3.79% ±4.93%
 url/legacy-vs-whatwg-url-parse.js n=100000 method='whatwg' type='file'              ***    -13.10 %       ±2.62% ±3.49% ±4.55%
 url/legacy-vs-whatwg-url-parse.js n=100000 method='whatwg' type='idn'               ***    -13.19 %       ±3.37% ±4.49% ±5.84%
 url/legacy-vs-whatwg-url-parse.js n=100000 method='whatwg' type='javascript'        ***    -19.30 %       ±2.03% ±2.71% ±3.52%
 url/legacy-vs-whatwg-url-parse.js n=100000 method='whatwg' type='long'              ***     -6.29 %       ±3.38% ±4.51% ±5.89%
 url/legacy-vs-whatwg-url-parse.js n=100000 method='whatwg' type='percent'           ***    -17.02 %       ±3.22% ±4.29% ±5.58%
 url/legacy-vs-whatwg-url-parse.js n=100000 method='whatwg' type='short'             ***    -13.14 %       ±3.11% ±4.14% ±5.39%
 url/legacy-vs-whatwg-url-parse.js n=100000 method='whatwg' type='ws'                ***    -14.13 %       ±2.30% ±3.06% ±3.98%

@joyeecheung
Copy link
Member Author

Added a type check for the argument of common.bakeUrlData

CI: https://ci.nodejs.org/job/node-test-pull-request/18534/

@joyeecheung
Copy link
Member Author

Ping @nodejs/url @nodejs/performance can I have some reviews please? As the data in #24302 (comment) demonstrates, this should not affect the accuracy of the benchmarks while providing more options for the data set.

Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@joyeecheung joyeecheung added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Nov 19, 2018
@joyeecheung
Copy link
Member Author

@Trott
Copy link
Member

Trott commented Nov 19, 2018

Since this affects benchmarks, let's run the benchmark CI job on it: https://ci.nodejs.org/job/node-test-commit-custom-suites/766/

@joyeecheung
Copy link
Member Author

joyeecheung commented Nov 19, 2018

Landed in 5f25dd1...8450d11, thanks!

joyeecheung added a commit that referenced this pull request Nov 19, 2018
This patch:

- Introduces `common.bakeUrlData` which can be used to pre-generate
  the data set for the URL benchmarks to loop through instead of
  looping over a constant.
- Add the option to use WPT data in benchmarks for better diversity
  in the input
- Add the option to benchmark URL parsing with base URLs (whatwg only)
- Moves the data in `benchmark/fixtures/url-inputs.js` to
  `benchmark/common.js`

PR-URL: #24302
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
joyeecheung added a commit that referenced this pull request Nov 19, 2018
This patch adds the option in the create-clientrequest benchmark
to accept URL inputs (as strings or as URL objects) so we can
measure the impact of URL parsing in a more sophisticated use case.

PR-URL: #24302
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
targos pushed a commit that referenced this pull request Nov 19, 2018
This patch:

- Introduces `common.bakeUrlData` which can be used to pre-generate
  the data set for the URL benchmarks to loop through instead of
  looping over a constant.
- Add the option to use WPT data in benchmarks for better diversity
  in the input
- Add the option to benchmark URL parsing with base URLs (whatwg only)
- Moves the data in `benchmark/fixtures/url-inputs.js` to
  `benchmark/common.js`

PR-URL: #24302
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
targos pushed a commit that referenced this pull request Nov 19, 2018
This patch adds the option in the create-clientrequest benchmark
to accept URL inputs (as strings or as URL objects) so we can
measure the impact of URL parsing in a more sophisticated use case.

PR-URL: #24302
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
rvagg pushed a commit that referenced this pull request Nov 28, 2018
This patch:

- Introduces `common.bakeUrlData` which can be used to pre-generate
  the data set for the URL benchmarks to loop through instead of
  looping over a constant.
- Add the option to use WPT data in benchmarks for better diversity
  in the input
- Add the option to benchmark URL parsing with base URLs (whatwg only)
- Moves the data in `benchmark/fixtures/url-inputs.js` to
  `benchmark/common.js`

PR-URL: #24302
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
rvagg pushed a commit that referenced this pull request Nov 28, 2018
This patch adds the option in the create-clientrequest benchmark
to accept URL inputs (as strings or as URL objects) so we can
measure the impact of URL parsing in a more sophisticated use case.

PR-URL: #24302
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
@BridgeAR BridgeAR mentioned this pull request Dec 5, 2018
4 tasks
@codebytere
Copy link
Member

@joyeecheung is this something you think should be backported to v10.x? I've added the label, but feel free to remove it!

refack pushed a commit to refack/node that referenced this pull request Jan 14, 2019
This patch:

- Introduces `common.bakeUrlData` which can be used to pre-generate
  the data set for the URL benchmarks to loop through instead of
  looping over a constant.
- Add the option to use WPT data in benchmarks for better diversity
  in the input
- Add the option to benchmark URL parsing with base URLs (whatwg only)
- Moves the data in `benchmark/fixtures/url-inputs.js` to
  `benchmark/common.js`

PR-URL: nodejs#24302
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
refack pushed a commit to refack/node that referenced this pull request Jan 14, 2019
This patch adds the option in the create-clientrequest benchmark
to accept URL inputs (as strings or as URL objects) so we can
measure the impact of URL parsing in a more sophisticated use case.

PR-URL: nodejs#24302
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. benchmark Issues and PRs related to the benchmark subsystem. http Issues or PRs related to the http subsystem. querystring Issues and PRs related to the built-in querystring module. test Issues and PRs related to the tests. url Issues and PRs related to the legacy built-in url module. whatwg-url Issues and PRs related to the WHATWG URL implementation.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants