Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding CI to release workflow to create a develop baseline performance on Bencher #3627

Merged
merged 8 commits into from
Sep 13, 2023

Conversation

Robertorosmaninho
Copy link
Collaborator

This modification is necessary to create a baseline for newer branches to compare the performance of their changes against what we already have in develop.

This modification is necessary to create and populate the develop's branch as the baseline for newer branches.
This script post_results_to_develop.py works by getting the last benchmarks reports of the last branch merged into develop, aggregating them in one file, and posting them on the develop branch of Bencher.

Hopefully, this is a temporary fix, as this was already accepted as a bug/possible feature by Bencher's maintainer bencherdev/bencher#187.

This PR, unlike #3625, doesn't need to rerun all Performance Tests on the develop or master branch.

@Robertorosmaninho Robertorosmaninho requested a review from a team as a code owner September 7, 2023 20:06
@rv-jenkins rv-jenkins changed the base branch from master to develop September 7, 2023 20:07
@Robertorosmaninho Robertorosmaninho self-assigned this Sep 7, 2023
.github/workflows/develop.yml Show resolved Hide resolved
.github/workflows/develop.yml Outdated Show resolved Hide resolved
Copy link
Contributor

@radumereuta radumereuta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something is not clear for me.
Where do you store the results?
I would like to avoid modifying the current PR and/or adding new commits to develop after a PR was merged.
I don't think this is the case here, but I don't understand exactly how this works.

@Robertorosmaninho
Copy link
Collaborator Author

Robertorosmaninho commented Sep 13, 2023

Something is not clear for me.
Where do you store the results?

We're currently storing the results in the Bencher cloud! It's free, and we don't have to worry about keeping a docker container to use and maintain as a server.

I would like to avoid modifying the current PR and/or adding new commits to develop after a PR was merged.
I don't think this is the case here, but I don't understand exactly how this works.

Here, we aren't modifying the PR or adding something to the develop branch after the PR is merged. Instead, we're just:

  1. querying the name of the last PR merged (using the last commit on develop) using GIT API and then querying the results of the last execution of Performance Tests on this PR using the Bencher API.
  2. This result is then POSTED in the develop branch data graph on the Bencher DB using the Bencher API as well.

Doing so, we don't have to rerun the Performance Tests again to collect the results for the develop branch.

After this PR, develop will have its own graph on Bencher that will work as a baseline for other PRs.

Copy link
Contributor

@radumereuta radumereuta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @Robertorosmaninho that clarifies things for me.
lgtm

@rv-jenkins rv-jenkins merged commit d339b10 into develop Sep 13, 2023
@rv-jenkins rv-jenkins deleted the post-bencher-results-to-develop branch September 13, 2023 14:25
rv-jenkins pushed a commit that referenced this pull request Sep 14, 2023
Follow up to #3627
This fix is necessary due to the lack of dependencies such as `python3`,
`curl`, and `bencher` on the test container:
[error](https://github.com/runtimeverification/k/actions/runs/6173858635/job/16757140239).


Running this container is lighter than using the `with-docker` version
of CI, and we can pass the various environment variables needed by
GitHub and Bencher API!
@Baltoli Baltoli mentioned this pull request Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants