Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update --deps-list format #25

Merged
merged 3 commits into from
Jul 28, 2023
Merged

Update --deps-list format #25

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
883a2a7
update: change --deps-list format
skyleaworlder Jul 26, 2023
dc2043d
feat: update parse_deps_list to support multiple sets of dependencies
skyleaworlder Jul 27, 2023
6d28bb7
refact: change runbenchmarks-cli.jl function, support multiple sets o…
skyleaworlder Jul 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ For specification, `Dependencies List` is a single string that simulates an arra
e.g.

```shell
> DEPS_LIST="https://github.com/FluxML/NNlib.jl#backports-0.8.21,https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test;Flux,Flux@0.13.12"
> DEPS_LIST="https://github.com/FluxML/NNlib.jl#backports-0.8.21,Flux;https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test,Flux@0.13.12"
> julia --project=benchmark benchmark/runbenchmarks-cli.jl --deps-list=$DEPS_LIST
```

Expand Down Expand Up @@ -93,7 +93,7 @@ More precisely, the granularity of the element of `Enabled Parts` and `Disabled
e.g.

```shell
> DEPS_LIST="https://github.com/FluxML/NNlib.jl#backports-0.8.21,https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test;Flux,Flux@0.13.12"
> DEPS_LIST="https://github.com/FluxML/NNlib.jl#backports-0.8.21,Flux;https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test,Flux@0.13.12"
> # Only Flux-MLP and all NNlib
> julia --project=benchmark benchmark/runbenchmarks-cli.jl --enable="flux(mlp);nnlib" --deps-list=$DEPS_LIST
> # All benchmarks except Flux, NNlib-gemm and NNlib-activations
Expand Down
74 changes: 21 additions & 53 deletions benchmark/runbenchmarks-cli.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,63 +1,31 @@
using Pkg

###########################################################################

Pkg.develop(PackageSpec(path = ENV["PWD"]))
using FluxMLBenchmarks

parsed_args = parse_commandline()
retune_arg = parsed_args["retune"]
retune_arg || get_tuning_json()
enable_arg = parsed_args["enable"]
disable_arg = parsed_args["disable"]
enable_arg, disable_arg = parsed_args["enable"], parsed_args["disable"]
enabled_benchmarks = parse_enabled_benchmarks(enable_arg, disable_arg)

deps_list = parsed_args["deps-list"]
baseline_fluxml_deps, target_fluxml_deps = parse_deps_list(deps_list)
time_setup_fluxml_env = @elapsed setup_fluxml_env(baseline_fluxml_deps)
@info "TIME: setup FluxML benchmarking environment (baseline) cost $time_setup_fluxml_env"

using BenchmarkTools
using PkgBenchmark
group_baseline = benchmarkpkg(
dirname(@__DIR__),
BenchmarkConfig(
env = merge(
Dict("JULIA_NUM_THREADS" => get(ENV, "JULIA_NUM_THREADS", "1")),
enabled_benchmarks
)
),
resultfile = joinpath(@__DIR__, "result-baseline.json")
)

teardown()

###########################################################################

Pkg.develop(PackageSpec(path = ENV["PWD"]))
using FluxMLBenchmarks

time_setup_fluxml_env = @elapsed setup_fluxml_env(target_fluxml_deps)
@info "TIME: setup FluxML benchmarking environment (target) cost $time_setup_fluxml_env"

using BenchmarkTools
using PkgBenchmark
group_target = benchmarkpkg(
dirname(@__DIR__),
BenchmarkConfig(
env = merge(
Dict("JULIA_NUM_THREADS" => get(ENV, "JULIA_NUM_THREADS", "1")),
enabled_benchmarks
)
),
resultfile = joinpath(@__DIR__, "result-target.json"),
)

teardown()

###########################################################################

judgement = judge(group_target, group_baseline)
report_md = markdown_report(judgement)
write(joinpath(@__DIR__, "report.md"), report_md)
display_markdown_report(report_md)
parsed_deps_list = parse_deps_list(deps_list)
for (i, deps) in enumerate(parsed_deps_list)
time_setup_fluxml_env = @elapsed setup_fluxml_env(deps)
@info "($i) TIME: setup FluxML benchmarking environment cost $time_setup_fluxml_env"

using PkgBenchmark: benchmarkpkg, BenchmarkConfig
time_run_benchmarks = @elapsed begin benchmarkpkg(
dirname(@__DIR__),
BenchmarkConfig(
env = merge(
Dict("JULIA_NUM_THREADS" => get(ENV, "JULIA_NUM_THREADS", "1")),
enabled_benchmarks
)
),
resultfile = joinpath(@__DIR__, "result-$i.json")
) end
@info "($i) TIME: run benchmarks cost $time_run_benchmarks"

teardown()
end
41 changes: 21 additions & 20 deletions src/env_utils.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,30 +229,31 @@ end


"""
parse_deps_list(deps_list::String)::Tuple{Vector{Dependency}, Vector{Dependency}}
parse_deps(deps)::Vector{Dependency}

is used to parse command argument, "deps-list", to 2 sets of dependencies,
which means parse_deps_list can support difference of multiple dependencies.
Each element separated by a semicolon. Each element consists of two parts:
the first part is baseline dependency, and the second part is target.
is used to parse dependencies.

* deps: suggested to be a string, can be `--target` or `--baseline`

e.g. deps can be like "NNlib,Flux" or
"https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test,Flux#0.13.12"
"""
parse_deps(deps) = map(dep -> Dependency(string(dep)), split(deps, ","))


"""
parse_deps_list(deps_list::String)::Tuple

is used to parse command argument, "deps-list" represents multiple sets of dependencies.
Each element separated by a semicolon. Each element consists of any number of deps.
Now, deps_list only supports FluxML packages.

e.g. deps_list can be
"NNlib,https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test;Flux,Flux#0.13.12"
"""
function parse_deps_list(deps_list::String)::Tuple{Vector{Dependency}, Vector{Dependency}}
dep_pairs = filter(
dep_pair_vec -> length(dep_pair_vec) == 2,
map(
dep_pair -> split(dep_pair, ","),
split(deps_list, ";")))
baseline_deps = map(
baseline_dep -> Dependency(string(baseline_dep)),
map(x -> x[1], dep_pairs))
target_deps = map(
target_dep -> Dependency(string(target_dep)),
map(x -> x[2], dep_pairs))
return (baseline_deps, target_deps)
"NNlib,Flux;https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test,Flux#0.13.12"
"""
function parse_deps_list(deps_list::String)::Tuple
parsed_deps_list = [parse_deps(deps) for deps in split(deps_list, ";")]
return Tuple(parsed_deps_list)
end


Expand Down
2 changes: 1 addition & 1 deletion test/env_utils_test.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
end

@testset "parse deps list" begin
deps_list = "NNlib,https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test;Flux,Flux@0.13.12"
deps_list = "NNlib,Flux;https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test,Flux@0.13.12"
baseline_deps, target_deps = parse_deps_list(deps_list)
@test (length(baseline_deps) == 2 &&
baseline_deps[1].name == "NNlib" &&
Expand Down