Continous Benchmarking Action #1137

JuanPedroGHM · 2023-03-31T14:22:18Z

Description

Enables performance tracking of HeAT by using Action for Continuous Benchmarking

Issue/s resolved: #

Changes proposed:

Moved old benchmarks to benchmarks/old
Created new benchmarks meant to run on the action inside of benchmarks/cb
Added perun as an optional dependency
CB action runs on every push of a pull requests with the PR talk tag
Introduced dedicated create_spherical_dataset method in the utils module that can be used in unit tests and benchmarks

Type of change

New Github workflow for pull requests added.
New benchmarks organization

Performance

perun works with MPI, and won't print duplicate results when running on multiple processes.

Due Diligence

All split configurations tested
Multiple dtypes tested in relevant functions
Documentation updated (if needed)
Title of PR is suitable for corresponding CHANGELOG entry

Does this change modify the behaviour of other functions? If so, which?

no

ghost · 2023-03-31T14:24:08Z

👇 Click on the image for a new way to code review

Legend

github-actions · 2023-03-31T14:26:33Z

Thank you for the PR!

github-actions · 2023-03-31T14:39:09Z

Thank you for the PR!

codecov · 2023-03-31T14:49:28Z

Codecov Report

Merging #1137 (7fe7a21) into main (9f6b2eb) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1137      +/-   ##
==========================================
+ Coverage   91.79%   91.81%   +0.01%     
==========================================
  Files          72       73       +1     
  Lines       10497    10518      +21     
==========================================
+ Hits         9636     9657      +21     
  Misses        861      861

Flag	Coverage Δ
unit	`91.81% <100.00%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
heat/utils/data/__init__.py	`100.00% <100.00%> (ø)`
heat/utils/data/spherical.py	`100.00% <100.00%> (ø)`

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2023-03-31T14:49:32Z

Thank you for the PR!

github-actions · 2023-03-31T15:03:26Z

Thank you for the PR!

github-actions · 2023-03-31T15:12:05Z

Thank you for the PR!

github-actions · 2023-04-04T08:51:07Z

Thank you for the PR!

github-actions · 2023-04-04T09:04:58Z

Thank you for the PR!

github-actions · 2023-04-04T09:11:38Z

Thank you for the PR!

github-actions · 2023-04-04T09:22:43Z

Thank you for the PR!

github-actions · 2023-04-04T09:32:08Z

Thank you for the PR!

github-actions · 2023-04-04T13:17:42Z

Thank you for the PR!

github-actions · 2023-04-04T13:50:54Z

Thank you for the PR!

github-actions · 2023-04-05T08:42:17Z

Thank you for the PR!

github-actions · 2023-04-05T09:14:28Z

Thank you for the PR!

mrfh92

Review for PR #1137 "Continuous Benchmarking Action"

This PR adds the possibility of continuous performance monitoring to Heat and therefore it is a highly valuable contribution to the overall functionality.

As far as I understand, there are three main changes coming along with this PR: First, those things required for the actual continuous benchmarking (including examples covering clustering and linear algebra); second, the "old" benchmarks have been moved to make place for the "new" continuous benchmarks; third, the unit tests for clustering have been cleaned up by introducing a dedicated function for creating the spherical test data set that now can also be used for benchmarking purposes.

As far as I can judge, the code is correct and implements the desired functionality. Since the CI runs through, I recommend merging.

… 'run bench' comment

github-actions · 2023-04-18T09:07:02Z

Thank you for the PR!

JuanPedroGHM · 2023-04-18T09:07:34Z

run bench

github-actions · 2023-04-18T11:35:43Z

Thank you for the PR!

github-actions · 2023-04-18T12:45:33Z

Thank you for the PR!

…d 'old' benchmarking folder to '2020'

github-actions · 2023-04-24T08:48:01Z

Thank you for the PR!

github-actions · 2023-04-24T08:50:25Z

Thank you for the PR!

github-actions · 2023-04-24T09:59:51Z

Thank you for the PR!

github-actions · 2023-04-26T04:06:52Z

Thank you for the PR!

ClaudiaComito

Thanks a lot @JuanPedroGHM , this is really helpful.

One general comment: it isn't obvious to me how to view the benchmark results once the Action has run.

Otherwise I only have tiny changes requests and am looking forward to merge. Thanks a lot!

ClaudiaComito · 2023-04-26T07:14:24Z

.github/workflows/benchmark_main.yml

+        uses: actions/checkout@v3
+      - name: Setup MPI
+        uses: mpi4py/setup-mpi@v1
+      - name: Use Python 3.8


shall we take 3.9 or 3.10?

I'll change both of them to python 3.10

ClaudiaComito · 2023-04-26T07:15:12Z

.github/workflows/benchmark_pr.yml

+      - name: Use Python 3.8
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.8 # Perun only supports 3.8 and ahead


3.9 or 3.10?

About the link to the benchmarks, I will add a new badge to the README that points to the plots.

ClaudiaComito · 2023-04-26T07:19:00Z

.github/workflows/benchmark_pr.yml

+jobs:
+  benchmark-pr:
+    name: Benchmark PR
+    if: contains(github.event.pull_request.labels.*.name, 'PR talk')


Let's have a dedicated label, "benchmark PR" or something? Would make it more intuitive esp. for future contributors.

…pointing to the benchmarks, changed trigger tag

github-actions · 2023-04-26T13:43:00Z

Thank you for the PR!

github-actions · 2023-04-26T13:47:57Z

Thank you for the PR!

ClaudiaComito

Wonderful, thanks a lot @JuanPedroGHM !

JuanPedroGHM mentioned this pull request Mar 31, 2023

Ginko Performance Explorer Support #1036

Closed

1 task

JuanPedroGHM added the benchmarking label Mar 31, 2023

JuanPedroGHM self-assigned this Mar 31, 2023

ClaudiaComito requested review from ClaudiaComito, Markus-Goetz, mtar and mrfh92 April 17, 2023 09:11

mrfh92 previously approved these changes Apr 18, 2023

View reviewed changes

JuanPedroGHM added 6 commits April 18, 2023 10:19

ci: continous benchmarking action

637c259

fix pytorch install command

24b8d77

fixed action python version

0ed8278

added github token to benchmark action

29d8f35

test message worst performance

a2c7d2a

ci: test gh-pages results from cb action

11ca920

ci: split cb workflow into a main and pr. pr workflow triggers with a…

dee3a77

… 'run bench' comment

JuanPedroGHM dismissed mrfh92’s stale review via dee3a77 April 18, 2023 09:02

JuanPedroGHM force-pushed the features/continuous-benchmarking branch from d16523f to dee3a77 Compare April 18, 2023 09:02

ci: bench pr runs after a review is requested

4c06c12

ci: bench now only triggers on pull requests with 'PR talk' tag

a40c40b

JuanPedroGHM added the PR talk label Apr 18, 2023

JuanPedroGHM added 2 commits April 24, 2023 10:42

ci: reshape benchmark, removed cronjob from pytorch workflows, rename…

a7baabd

…d 'old' benchmarking folder to '2020'

mend

2888cfc

fix: missing import in cb main

be9cd0a

Merge branch 'main' into features/continuous-benchmarking

1eda895

ClaudiaComito requested changes Apr 26, 2023

View reviewed changes

ci: changed benchmark python version to 3.10, added shield in readme …

2259e9b

…pointing to the benchmarks, changed trigger tag

JuanPedroGHM added the benchmark PR label Apr 26, 2023

fix: incorrect python version param

7fe7a21

ClaudiaComito approved these changes Apr 27, 2023

View reviewed changes

ClaudiaComito added this to the 1.3.0 milestone Apr 27, 2023

ClaudiaComito added workflows and removed PR talk labels Apr 27, 2023

ClaudiaComito merged commit 02d2492 into main Apr 27, 2023

ClaudiaComito deleted the features/continuous-benchmarking branch April 27, 2023 03:11

Continous Benchmarking Action #1137

Continous Benchmarking Action #1137

Conversation

JuanPedroGHM commented Mar 31, 2023 • edited Loading

Description

Changes proposed:

Type of change

Performance

Due Diligence

Does this change modify the behaviour of other functions? If so, which?

ghost commented Mar 31, 2023 • edited by ghost Loading

Legend

github-actions bot commented Mar 31, 2023

github-actions bot commented Mar 31, 2023

codecov bot commented Mar 31, 2023 • edited Loading

Codecov Report

github-actions bot commented Mar 31, 2023

github-actions bot commented Mar 31, 2023

github-actions bot commented Mar 31, 2023

github-actions bot commented Apr 4, 2023

github-actions bot commented Apr 4, 2023

github-actions bot commented Apr 4, 2023

github-actions bot commented Apr 4, 2023

github-actions bot commented Apr 4, 2023

github-actions bot commented Apr 4, 2023

github-actions bot commented Apr 4, 2023

github-actions bot commented Apr 5, 2023

github-actions bot commented Apr 5, 2023

mrfh92 left a comment • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Apr 18, 2023

JuanPedroGHM commented Apr 18, 2023

github-actions bot commented Apr 18, 2023

github-actions bot commented Apr 18, 2023

github-actions bot commented Apr 24, 2023

github-actions bot commented Apr 24, 2023

github-actions bot commented Apr 24, 2023

github-actions bot commented Apr 26, 2023

ClaudiaComito left a comment

Choose a reason for hiding this comment

ClaudiaComito Apr 26, 2023

Choose a reason for hiding this comment

JuanPedroGHM Apr 26, 2023

Choose a reason for hiding this comment

ClaudiaComito Apr 26, 2023

Choose a reason for hiding this comment

JuanPedroGHM Apr 26, 2023

Choose a reason for hiding this comment

ClaudiaComito Apr 26, 2023

Choose a reason for hiding this comment

JuanPedroGHM Apr 26, 2023

Choose a reason for hiding this comment

github-actions bot commented Apr 26, 2023

github-actions bot commented Apr 26, 2023

ClaudiaComito left a comment

Choose a reason for hiding this comment

JuanPedroGHM commented Mar 31, 2023 •

edited

Loading

ghost commented Mar 31, 2023 •

edited by ghost

Loading

codecov bot commented Mar 31, 2023 •

edited

Loading

mrfh92 left a comment •

edited

Loading