Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve & compute statistical significance #40

Open
10 tasks
vweevers opened this issue Nov 19, 2022 · 0 comments
Open
10 tasks

Improve & compute statistical significance #40

vweevers opened this issue Nov 19, 2022 · 0 comments
Labels
benchmark Requires or pertains to benchmarking

Comments

@vweevers
Copy link
Member

vweevers commented Nov 19, 2022

I'm getting to the point where changes in e.g. abstract-level result in benchmark differences that are too small to say anything meaningful about it. Which is a good sign, but it means that the benchmarks must become more precise to still be useful. Rough plan:

  • Measure timer resolution, aka average smallest measurable time
  • Determine minimum duration of benchmark by Math.max(resolutionInSeconds() / 2 / 0.01, 0.05) * 1e3
  • Let iterations be 1
  • Optionally run benchmark for warmup: fn(); fn(); eval('%OptimizeFunctionOnNextCall(fn)'); fn()
  • Run benchmark, which should call a function iterations amount of times
  • Optionally subtract time spent on GC
  • If needed, increase iterations to satisfy minimum duration, and repeat
  • If minimum duration is met, record the duration in a histogram
  • Optionally also factor in histogram.minimumSize()
  • Then summarize a la 1ebed31
@vweevers vweevers added this to Level Nov 19, 2022
@vweevers vweevers moved this to Backlog in Level Nov 19, 2022
@vweevers vweevers added the benchmark Requires or pertains to benchmarking label Feb 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
benchmark Requires or pertains to benchmarking
Projects
Status: Backlog
Development

No branches or pull requests

1 participant