Benchmarks

[kazimuth]

It would be cool to have some benchmarks for different system components, to make it easier to avoid causing performance regressions. It seems like hecs has some, which is cool.

https://github.com/marketplace/actions/continuous-benchmark could also be fun to set up, it gives you graphs of performance over time like this. Not sure how much time that would add to CI builds though, I haven't used GitHub Actions much. Having a graph that measured average incremental-recompile time could be very helpful in preventing that from creeping up.

[cart]

Thats a fantastic idea. I also really want performance repression tests and would love to seem them in any form.

At the rate that ive been committing, i already capped the free github actions limit back when i had it validate every commit. Now it runs once daily (and for prs) but im still pretty tightly bound. I dont think we could fit perf tests on top. For anything gpu-bound we would need a box with a reasonable gpu, which im guessing we dont get by default.

So i think CI probably isn't feasible until we find funding, but I fully support building up benchmarks to run locally / compare branches. And can always plug those in to CI and cross our fingers 😄

[alice-i-cecile]

Work is in progress on this, see #4225. Closing this out now as complete-ish.