Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue 1044 auto benchmarking #1048

Merged
merged 69 commits into from
Mar 27, 2021
Merged

Issue 1044 auto benchmarking #1048

merged 69 commits into from
Mar 27, 2021

Conversation

nerdvegas
Copy link
Contributor

Closes #1044

  • A new tool, rez-benchmark, was added
  • This runs a predetermined set of resolves over a provided package repo
  • On release, this tool is used in a new benchmark.yaml workflow
  • The workflow stores its results as artifacts
  • These are stored in rez's github repo, and are summarised and appended to a RESULTS.md file
  • Any change in resolves (compared to the previous benchmark) is detected, and will cause the workflow to fail

ajohns and others added 30 commits March 16, 2021 13:47
- added new data/ dir
5e1419e 
- moved test data, updated tests
- some cleanup in support/
-updated import test
538b7e6 
-updated new benchmark test to point at benchmark data
wip
b6984bd
md test
7d0bf95
added benchmark workflow, first pass
b495213
wip
24164f0
readying for new layout, where workflow writes out to /metrics
76e5fca
fixed lurking py3 bug
be526f5
wip
65f8ab4
debugging
cb593b0
debugging 2
7cc84e6
debugging 3
c447b1b
debug 4
27e1680
debug 5
ffec457
debug 6
cdf0aa9
Generated from GitHub benchmark Workflow
c7fb6fc 
See https://github.com/nerdvegas/rez/runs/run2
debug 7
5e60a41
debug 8
97c32c6
Generated from GitHub benchmark Workflow
db39631 
See https://github.com/nerdvegas/rez/runs/run2
debug 9
4ec736c
Generated from GitHub benchmark Workflow
a615b3f 
See https://github.com/nerdvegas/rez/runs/run2
debug 10
dc198c6
Generated from GitHub benchmark Workflow
589b57e 
See https://github.com/nerdvegas/rez/runs/run2
debug 11
6f558cb
Merge branch 'issue_1044-auto-benchmarking' of github.com:nerdvegas/r…
446d4d4 
…ez into issue_1044-auto-benchmarking
Generated from GitHub benchmark Workflow
c632b00 
See https://github.com/nerdvegas/rez/runs/run3
debug 12
3cefedb
Merge branch 'issue_1044-auto-benchmarking' of github.com:nerdvegas/r…
c4b4b81 
…ez into issue_1044-auto-benchmarking
Generated from GitHub benchmark Workflow
d48b7b5 
See https://github.com/nerdvegas/rez/runs/run3
debug 13
4274990
github.com/nerdvegas and others added 29 commits March 23, 2021 05:16
Generated from GitHub benchmark Workflow
06a491b 
See https://github.com/nerdvegas/rez/runs/run3
Generated from GitHub benchmark Workflow
a944412 
See https://github.com/nerdvegas/rez/runs/run3
workflow comment
091b786
Merge branch 'issue_1044-auto-benchmarking' of github.com:nerdvegas/r…
26e566f 
…ez into issue_1044-auto-benchmarking
-added SOLVER_VERSION
7193de7 
-added workflow step to validate benchmark against most recent
try no parallel, got '[error] The operation was canceled', not sure w…
7f51af5 
…hy yet
fix for empty artifacts dir
baee973
Generated from GitHub benchmark Workflow
0beebe7
Generated from GitHub benchmark Workflow
7cb3cc9
testing benchmark validation
c36d2df
fixed path bug in validation
711db33
more readable 'resolves differ' message
71e05b9
report smallest mismatched solve result
71c595d
reenable all tests
26c0b85
-a change that hopfully fixes solver diff issue
b52ef80 
-I suspect it's change in set ordering in py2/3 causing this
Generated from GitHub benchmark Workflow
fe87385
Generated from GitHub benchmark Workflow
9a9637e
added more info to benchmark workflow output
7c002c1
Generated from GitHub benchmark Workflow
076c6a1
Generated from GitHub benchmark Workflow
a36ff52
testing
40d9c9f
Generated from GitHub benchmark Workflow
2f9458f
Generated from GitHub benchmark Workflow
74d2344
finished, tidying up
a9d1ec9
benchmark runs on release only
745e770
removed support/benchmarking
e0cf8de
Generated from GitHub benchmark Workflow
54b3777
Generated from GitHub benchmark Workflow
3dfdf3e
removed prev benchmark results, why were these generated?
1fa9e8a
@nerdvegas nerdvegas merged commit 3812260 into master Mar 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

automatically run benchmarking
1 participant
@nerdvegas