feat: run benchmarks in non-bench targets #85
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As discussed in and closes #84. I also made it use the
bench
profile by default when--benchmarks
is supplied, and add the--bench
argument to the target args likecargo bench
does (otherwise it runs unit tests instead).For reference, you can (optionally) specify which benchmarks to run by adding them as target args:
Otherwise it'll run all of the benchmarks in the target.
Limitations:
cargo instruments -t time --benchmarks
to fail because it'll build multiple targets and it expects a single one (like if--benchmarks
hadn't been supplied). AFAIK,cargo instruments
can only run one target, so it has no analogous operation tocargo bench
running benchmarks in multiple targets by default, so maybe this is OK. An alternative would be to run benchmarks in the main target by default, but this might be unexpected for users ofcargo bench
. Workaround is to specify a single target likecargo instruments -t time --benchmarks --bin <crate_name>
. Might be nice to point users in this direction in the error message.--benchmarks
since everywhere else incargo
"bench" is used, but overloading the--bench
option seems more confusing. I could see an argument for--benchmark
as it looks like an action, instead of using two plurals in a row ("instruments", "benchmarks"). But--benchmarks
makes it clear that it'll run all of them in the chosen target.cargo instruments --bench
without a value, the error message doesn't ask them to use--benchmarks
instead. But I did add a note about it in the--help
message for--bench
.--bench
to the target args likecargo bench
does, but I haven't checked thecargo bench
source code to see if there are exceptions to this behavior (say, when the default test harness is disabled).cargo instruments --bench
should imply--benchmarks
? We'd probably want to use thebench
profile by default, for example. I'm not using non-default test harnesses or separate bench targets though, so I haven't tested what the best behavior here should be.