Run Benchmark Tests on a Given Branch

[stanbrub]

High Level Problem

• Developer makes improvements on a branch to an existing DH operation (e.g. where)
• Developer wants to compare benchmarks for changes against existing nightly benchmarks
• For consistency, run benchmarks on the same server as the nightly builds

Questions

• Is this done in an adhoc way, without preserving the results?
• Since nightly is done against docker, how do we do a proper comparison to the branch?
• How automated is this? Do it from a github workflow?

[devinrsmith]

Yeah - the use-case is that we might want to have benchmarks as one of the deciding factors for deephaven-core PR approval (not necessarily for all PRs, but those specifically touting to be operational improvements that we should be able to benchmark).

It's easy enough for me to do ad-hoc "benchmarking" locally, but it would be nice if there was an official way.

[stanbrub]

So it sounds like this would be something driven by a github workflow. And we'd have to figure out things like how to make sure two tests don't run on the benchmark system or add bare metal provisioning into the workflow.