Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update --deps-list format #25

Merged
merged 3 commits into from
Jul 28, 2023

Conversation

skyleaworlder
Copy link
Collaborator

PR Checklist

  • Tests are added
  • Documentation, if applicable

Description

Close #24

Make --deps-list clearer, and step to arbitrary dependencies.

@darsnack
Copy link
Member

So ; separates different runs of the benchmark, and , separates dependencies within a run? Looks good to me.

@skyleaworlder
Copy link
Collaborator Author

So ; separates different runs of the benchmark, and , separates dependencies within a run? Looks good to me.

Yes. I found this would be much better.

And I also found if it's better, why can I support multiple dependencies for --baseline and --target... also mentioned in #24

image

@codecov-commenter
Copy link

Codecov Report

❗ No coverage uploaded for pull request base (main@3a7a617). Click here to learn what that means.
Patch has no changes to coverable lines.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #25   +/-   ##
=======================================
  Coverage        ?   36.89%           
=======================================
  Files           ?        4           
  Lines           ?      187           
  Branches        ?        0           
=======================================
  Hits            ?       69           
  Misses          ?      118           
  Partials        ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@skyleaworlder
Copy link
Collaborator Author

skyleaworlder commented Jul 27, 2023

In addition, this PR refines benchmark/runbenchmarks-cli.jl.

Now, we cannot judge two BenchmarkResults and generate the report any longer. But on the other hand, it supports to pass multiple sets of dependencies to run benchmarks, which means that the following operation will be feasible:

> # NOTICE! DEPS_LIST contains THREE sets of dependencies
> DEPS_LIST="https://github.com/FluxML/NNlib.jl#backports-0.8.21,Flux;https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test,Flux@0.13.12;NNlib,Flux"
>
> # Still only Flux-MLP
> # gonna generate 3 files: result-1.json, result-2.json and result-3.json
> julia --project=benchmark benchmark/runbenchmarks-cli.jl --enable="flux(mlp)" --deps-list=$DEPS_LIST

I have to merge this PR directly in the tomorrow morning, because this PR is related to another one to some extent. I still try to realize our previous discussions, and hopefully I can commit some code in another PR before our meeting.

@skyleaworlder skyleaworlder merged commit 44c4c54 into FluxML:main Jul 28, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

--deps-list is not suitable for extending
3 participants