Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support precise benchmark selection #19

Merged
merged 7 commits into from
Jul 19, 2023

Conversation

skyleaworlder
Copy link
Collaborator

PR Checklist

  • Tests are added
  • Documentation, if applicable

Description

More precisely, the granularity of the element of Enabled Parts and Disabled Parts is currently at the file-level, and supports two levels of files, which means that now our tool will recognize the name of each file in benchmark/benchmark and all the files under benchmark/benchmark/** before benchmarking.

Each top-level element in Enabled Parts and Disable Parts should be exactly the name of the file under benchmark/benchmark; each second-level element should be the name of the file under the dir that has the same name of top-level file.

e.g.

> DEPS_LIST="https://github.com/FluxML/NNlib.jl#backports-0.8.21,https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test;Flux,Flux@0.13.12"
> # Only Flux-MLP and all NNlib
> julia --project=benchmark benchmark/runbenchmarks-cli.jl --enable="flux:mlp;nnlib" --deps-list=$DEPS_LIST
> # All benchmarks except Flux, NNlib-gemm and NNlib-activations
> julia --project=benchmark benchmark/runbenchmarks-cli.jl --disable="flux;nnlib:gemm,activations" --deps-list=$DEPS_LIST
> # Only Flux
> julia --project=benchmark benchmark/runbenchmarks-cli.jl --enable="flux;nnlib" --disable="nnlib" --deps-list=$DEPS_LIST

@codecov-commenter
Copy link

Codecov Report

❗ No coverage uploaded for pull request base (main@775e2e3). Click here to learn what that means.
Patch has no changes to coverable lines.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #19   +/-   ##
=======================================
  Coverage        ?   51.70%           
=======================================
  Files           ?        3           
  Lines           ?      147           
  Branches        ?        0           
=======================================
  Hits            ?       76           
  Misses          ?       71           
  Partials        ?        0           

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@skyleaworlder
Copy link
Collaborator Author

I create this PR since this is an independent feature. I'll merge this PR without any checks, in order to continue coding based on this.

@skyleaworlder skyleaworlder merged commit 39072e0 into FluxML:main Jul 19, 2023
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants