Use Bencher to benchmark PRs and track metrics

[epompeii]

Would the diesel project be interested in a PR to start tracking benchmarks and catching performance regressions with Bencher?

I ran across @weiznich post on Reddit regarding diesel benchmarking: https://old.reddit.com/r/rust/comments/15h6rsv/crate_authors_how_do_you_prevent_performance/juqk4ft/

Is there anything that https://github.com/diesel-rs/metrics does not provide that you would like to have in a benchmark tracking tool?

[weiznich]

Can you explain what Bencher offers what's not covered by the current setup?

[weiznich]

Posting to PR's would be helpful, although I've mostly solved that by using action summaries. See this PR for an example. It's currently by design that we run benchmarks only for certain PR by manually applying labels. That shouldn't be changed due to having a limited amount of CI workers.

As for tracking data over time. We have a different job for that, that runs more benchmarks (also for other crates). It's important for me to have the resulting data open, therefore they are pushed to a git repository. In addition: From the examples on the bencher webpage it seems like you offer only a graph of results over time. It's nice to have that as interactive graph, but we are also interested in comparison graphs for the last run, like that one (Although that would be more useful if it was interactive as well).

To summarize what is written above: I feel that there are still features missing to move the benchmark over to bencher. Most notably:

• The ability to run the benchmarks only for certain PR based on tags
• Advanced analytic formats (boxcharts to compare values for a specific run + maybe data tables that show [min, max, media, mean] per group + diff to a reference value)

[epompeii]

Okay, that all makes sense.

The ability to run the benchmarks only for certain PR based on tags

I believe that this can be accomplished with Bencher in the same way you all do it now. The main change would be here:

diesel/.github/workflows/benches.yml

Line 54 in b51567f

run: cargo +stable bench --manifest-path diesel_bench/Cargo.toml --no-default-features --features "${{matrix.backend}}" -- --save-baseline changes

To using Bencher, something like this:

 run: bencher run --err 'cargo +stable bench --manifest-path diesel_bench/Cargo.toml --no-default-features --features "${{matrix.backend}}"'

You wouldn't need the following checkout and critcmp.

Advanced analytic formats (boxcharts to compare values for a specific run + maybe data tables that show [min, max, media, mean] per group + diff to a reference value)

This I totally get! I have a tracking issue for point-in-time comparisons (bencherdev/bencher#72).
In the mean time, would you be interested in a PR that also starts saving the metrics to Bencher in addition to https://github.com/diesel-rs/metrics ?
This is what rustls is about to start doing. The change would be somewhere around here:

diesel/.github/workflows/metrics.yml

Line 78 in b51567f

- name: Push metrics

Then you can have the best of both worlds while I work on adding new plots/tables.

It's important for me to have the resulting data open

For a public project like this one, anyone can query the API endpoint for the data on Bencher. No auth or signup required.
This is all the metrics for the past month of Bencher dogfooding its devel branch
I'm curious if you would like things to be anymore open than that?

[weiznich]

This I totally get! I have a tracking issue for point-in-time comparisons (bencherdev/bencher#72).
In the mean time, would you be interested in a PR that also starts saving the metrics to Bencher in addition to https://github.com/diesel-rs/metrics ?
rustls/rustls-bench-app#6. The change would be somewhere around here:

That issue does not completely cover what I would like to see as bar charts are different to the box plots that are currently used. Box plots cover the statistical distribution for each set of values, while a bar chart merely covers a single value per group.

For a public project like this one, anyone can query the API endpoint for the data on Bencher. No auth or signup required.
This is all the metrics for the past month of Bencher dogfooding its devel branch
I'm curious if you would like things to be anymore open than that?

I'm sorry, but I trust github more to not go away anytime soon than a new service like bencher. That's the reason I prefer to have the data just there in the repo. (Also migrating to bencher would loose the old data, right?) If it's possible to do both I'm open to go that way.

I think I would be open with having bencher as additional target for the results and keep both solutions in parallel for some time. After that we can see how well each solution works and decide how we want to continue there.

[epompeii]

That issue does not completely cover what I would like to see as bar charts are different to the box plots that are currently used.

Very true! I have updated the issue to include box plots as a part of it. I see that working in a similar fashion to the toggling of the x-axis in the current line plots (datetime <-> branch version), so it would toggle between: bar chart <-> box plot. Thank you for making sure I note the difference!

I'm sorry, but I trust github more to not go away anytime soon than a new service like bencher

Totally fair! I'd feel the same way.

Also migrating to bencher would loose the old data, right?

Bencher can import the old data. It's possible to backdate reports. I think we should hold off on that for now though.

I think I would be open with having bencher as additional target for the results and keep both solutions in parallel for some time. After that we can see how well each solution works and decide how we want to continue there.

Sounds good! I'll start working on a PR for tracking the master branch metrics.

[epompeii]

PR is up: #3849

Please, let me know what you think! 😃