Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance benchmark visibility #1275

Closed
tsandall opened this issue Mar 20, 2019 · 0 comments · Fixed by #1659
Closed

Improve performance benchmark visibility #1275

tsandall opened this issue Mar 20, 2019 · 0 comments · Fixed by #1659

Comments

@tsandall
Copy link
Member

We currently have automation that periodically runs the benchmark suite against the tip of master and the last stable release. The results are posted to http://opa-benchmark-results.s3-website-us-east-1.amazonaws.com/. This helps us identify performance regressions during development.

The script that generates the benchstat results takes about half an hour to run. It takes this long because we have to run the entire benchmark suite 2*N times (N times on tip and N times on latest release).

It would be nice if we could come up with a way to make the benchmark results more visible without introducing so much latency on the pre-merge check. Perhaps we could generate the benchstat results post-merge and then add a comment to the PR. This way developers would be reminded to review the results on each merge.

@tsandall tsandall self-assigned this Aug 21, 2019
tsandall added a commit to tsandall/opa that referenced this issue Aug 22, 2019
Fixes open-policy-agent#1275

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
tsandall added a commit that referenced this issue Aug 22, 2019
Fixes #1275

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant