DEV: Publishing benchmarks vs. trunk?

[wesm]

We'll want to develop a microbenchmark suite to compare pandas 2.0 perf versus 1.x, especially in microbenchmarks. What's the right tool for this? asv, vbench?

[jorisvandenbossche]

IMO certainly not vbench (unless someone wants to put time in developing, but even then). The benchmark suite already is ported over to asv, and I have a PR to clean them up (pandas-dev/pandas#14099, once that is finished, I would remove the vbench ones, we don't use them anymore).
But I have less of an idea how reliable it is for microbenchmarks (I think it is based on timeit, and I think there are arguments to be made that this is not really reliable for microbenchmarks)

[wesm]

I was thinking that ASV is the best choice -- as long as we can easily compare perf vs. a snapshot of master (e.g. we can use v0.19.0 as a baseline) so we can see how we're doing.