-
Notifications
You must be signed in to change notification settings - Fork 153
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding CI to release workflow to create a develop
baseline performance on Bencher
#3627
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Something is not clear for me.
Where do you store the results?
I would like to avoid modifying the current PR and/or adding new commits to develop after a PR was merged.
I don't think this is the case here, but I don't understand exactly how this works.
We're currently storing the results in the Bencher cloud! It's free, and we don't have to worry about keeping a docker container to use and maintain as a server.
Here, we aren't modifying the PR or adding something to the develop branch after the PR is merged. Instead, we're just:
Doing so, we don't have to rerun the After this PR, develop will have its own graph on Bencher that will work as a baseline for other PRs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @Robertorosmaninho that clarifies things for me.
lgtm
Follow up to #3627 This fix is necessary due to the lack of dependencies such as `python3`, `curl`, and `bencher` on the test container: [error](https://github.com/runtimeverification/k/actions/runs/6173858635/job/16757140239). Running this container is lighter than using the `with-docker` version of CI, and we can pass the various environment variables needed by GitHub and Bencher API!
This modification is necessary to create a baseline for newer branches to compare the performance of their changes against what we already have in develop.
This modification is necessary to create and populate the develop's branch as the baseline for newer branches.
This script
post_results_to_develop.py
works by getting the last benchmarks reports of the last branch merged into develop, aggregating them in one file, and posting them on the develop branch of Bencher.Hopefully, this is a temporary fix, as this was already accepted as a bug/possible feature by Bencher's maintainer bencherdev/bencher#187.
This PR, unlike #3625, doesn't need to rerun all
Performance Tests
on the develop or master branch.