Support selected benchmarks

[skyleaworlder]

PR Checklist

• Tests are added
• Documentation, if applicable

Description

Benchmarking always takes amount of time. In order to focus on the targets and reduce the time consumption of our benchmarking tool, the --enable and --disable options are used to specify the parts to be included and the parts to be excluded respectively.

> julia --project=benchmark benchmark/runbenchmarks-cli.jl  \
>   --enable=<ENABLED_PARTS> \
>   --disable=<DISABLED_PARTS> \
>   --deps-list=<Dependencies List>

For specification, Enabled Parts and Disabled Parts have the same format, which is a single string that simulates an array, with each element separated by a comma.

The granularity of the element of Enabled Parts and Disabled Parts is currently at the file-level, which means now our tool will recognize the name of each file in benchmark/benchmark before benchmarking and each element in Enabled Parts and Disable Parts should be exactly the name of those files.

--enable is used to specify the files that should be included, and by default (--enable not specified) all files in the benchmark/benchmark are included. --disable is used to specify the files that should be excluded, and the default value is an empty string.

e.g.

> DEPS_LIST="https://github.com/FluxML/NNlib.jl#backports-0.8.21,https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test;Flux,Flux@0.13.12"
> # Only Flux and NNlib
> julia --project=benchmark benchmark/runbenchmarks-cli.jl --enable="flux,nnlib" --deps-list=$DEPS_LIST
> # All benchmarks except Flux and NNlib
> julia --project=benchmark benchmark/runbenchmarks-cli.jl --disable="flux,nnlib" --deps-list=$DEPS_LIST
> # Only Flux
> julia --project=benchmark benchmark/runbenchmarks-cli.jl --enable="flux,nnlib" --disable="nnlib" --deps-list=$DEPS_LIST

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request base (main@aa136ae). Click here to learn what that means.
Patch has no changes to coverable lines.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #14   +/-   ##
=======================================
  Coverage        ?   46.92%           
=======================================
  Files           ?        3           
  Lines           ?      130           
  Branches        ?        0           
=======================================
  Hits            ?       61           
  Misses          ?       69           
  Partials        ?        0           

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.