From f4ee4da5a6d1148459482fc1ef24e6e348d3be4f Mon Sep 17 00:00:00 2001 From: Stefan Marr Date: Sat, 14 Jul 2018 15:56:15 +0100 Subject: [PATCH] Document all configuration options. Signed-off-by: Stefan Marr --- docs/config.md | 455 ++++++++++++++++++++++++++++++++++++- docs/extensions.md | 10 +- rebench/rebench-schema.yml | 8 +- 3 files changed, 464 insertions(+), 9 deletions(-) diff --git a/docs/config.md b/docs/config.md index 2fb68ceb..552cdf74 100644 --- a/docs/config.md +++ b/docs/config.md @@ -295,8 +295,461 @@ runs: execute_exclusively: false ``` +## Reporting - +Currently, [Codespeed] is the only supported system for continuous +performance monitoring. It is configured with the `reporting` key. + +**codespeed:** + +Send results to Codespeed for continuous performance tracking. +The settings define the project that is configured in Codespeed, and the +URL to which the results are reported. Codespeed requires more information, +but since these details depend on the environment, they are passed set on +the [command line](usage.md#continuous-performance-tracking). + +Example: + +```yaml +reporting: + codespeed: + project: MyVM + url: http://example.org/result/add/json/ +``` + +--- + +## Benchmark Suites + +Benchmark suites are named collections of benchmarks and settings that apply to +all of them. + +**gauge_adapter:** + +Name of the parser that interpreters the output of the benchmark harness. +For a list of supported options see the list of [extensions](extensions.md#available-harness-support). + +This key is mandatory. + +Example: + +```yaml +benchmark_suites: + ExampleSuite: + gauge_adapter: ReBenchLog +``` + +--- + +**command:** + +The command for the benchmark harness. It's going to be combined with the +VM's command line. Thus, it should instruct the VM which harness to use +and how to map the various parameters to the corresponding harness settings. + +It supports various format variables, including: + + - benchmark (the benchmark's name) + - cores (the number of cores to be used by the benchmark) + - input (the input variable's value) + - iterations (the number of iterations) + - variable (another variable's value) + - warmup (the number of iterations to be considered warmup) + +This key is mandatory. + +Example: + +```yaml +benchmark_suites: + ExampleSuite: + command: Harness %(benchmark)s --problem-size=%(input)s --iterations=%(iterations)s +``` + +--- + +**location:** + +The path to the benchmark harness. Execution use this location as +working directory. It overrides the location/path of a VM. + +Example: + +```yaml +benchmark_suites: + ExampleSuite: + location: ../benchmarks/ +``` + +--- + +**build:** + +The given string is executed by the system's shell and can be used to +build a benchmark suite. It is executed once before any benchmarks of the suite +are executed. If `location` is set, it is used as working directory. +Otherwise, it is the current working directory of ReBench. + +Example: + +```yaml +benchmark_suites: + ExampleSuite: + build: ./build-suite.sh +``` + +--- + +**description/desc:** + +The keys `description` and `desc` can be used to add a simple explanation of +the purpose of the suite. + +Example: + +```yaml +benchmark_suites: + ExampleSuite: + description: | + This is an example suite for this documentation. +``` + +--- + +**benchmarks:** + +The `benchmarks` key takes the list of benchmarks. Each benchmark is either a +simple name, or a name with additional properties. +See the section on [benchmark](#benchmark) for details. + +Example: + +```yaml +benchmark_suites: + ExampleSuite: + benchmark: + - Benchmark1 + - Benchmark2: + extra_args: "some additional arguments" +``` + +--- + +**run details and variables:** + +A benchmark suite can additional use the keys for [run details](#runs) and +[variables](#benchmark). +Thus, one can use: + +- `invocations` +- `iterations` +- `warmup` +- `min_iteration_time` +- `max_invocation_time` +- `parallel_interference_factor` +- `execute_exclusively` + +As well as: + +- input_sizes +- cores +- variable_values + +## Benchmark + +A benchmark can be define simply as a name. However, some times one might want +to define extra properties. + +**extra_args:** + +This extra argument is appended to the benchmark's command line. + +Example: + +```yaml +- Benchmark2: + extra_args: "some additional arguments" +``` + +--- + +**command:** + +ReBench will use this command instead of the name for the command line. + +Example: + +```yaml +- Benchmark2: + command: some.package.Benchmark2 +``` + +--- + +**codespeed_name:** + +A name used for this benchmark when sending data to Codespeed. +This gives more flexibility to keep Codespeed and these configurations or +source code details decoupled. + +Example: + +```yaml +- Benchmark2: + codespeed_name: "[peak] Benchmark2" +``` + +--- + +**input_sizes:** + +Many benchmark harnesses and benchmarks take an input size as a +configuration parameter. It might identify a data file, or some other +way to adjust the amount of computation performed. + +`input_sizes` expects a list, either as in the list notation below, or +in form of a sequence literal: `[small, large]`. + +Example: + +```yaml +- Benchmark2: + input_sizes: + - small + - large +``` + +--- + +**cores:** + +The number of cores to be used by the benchmark. +At least that's the original motivation for the variable. +In practice, it is more flexible and just another variable that can take +any list of strings. + +Example: + +```yaml +- Benchmark2: + cores: [1, 3, 4, 19] +``` + +--- + +**variable_values:** + +Another dimension by which the benchmark execution can be varied. +It takes a list of strings, or arbitrary values really. + +Example: + +```yaml +- Benchmark2: + variable_values: + - Sequential + - Parallel + - Random +``` + +--- + +**run details:** + +A benchmark suite can additional use the keys for [run details](#runs). + +--- + +## Virtual Machines + +The `virtual_machines` key defines the binaries and their settings to be used +to execute benchmarks. Each VM is a named set of properties. + +**path:** + +Path to the binary. If not given, it's up to the shell to find the binary. + +Example: + +```yaml +virtual_machines: + MyBin1: + path: . +``` + +--- + +**binary:** + +The name of the binary to be used. + +Example: + +```yaml +virtual_machines: + MyBin1: + binary: my-vm +``` + +--- + +**args:** + +The arguments given to the VM. They are given right after the binary. + +Example: + +```yaml +virtual_machines: + MyBin1: + args: --enable-assertions +``` + +--- + +**description and desc:** + +The keys `description` and `desc` can be used to document the purpose of the +VM specified. + +Example: + +```yaml +virtual_machines: + MyBin1: + desc: A simple example for testing. +``` + +--- + +**build:** + +The given string is executed by the system's shell and can be used to +build a VM. It is executed once before any benchmarks are executed with +the VM. If `path` is set, it is used as working directory. Otherwise, +it is the current working directory of ReBench. + +Example: + +```yaml +virtual_machines: + MyBin1: + build: | + make clobber + make +``` + +--- + +**run details and variables:** + +A VM can additional use the keys for [run details](#runs) and [variables](#benchmark) +(`input_sizes`, `cores`, `variable_values`). + +## Experiments + +Experiments combine virtual machines and benchmark suites. +They can be defined by listing suites to be used and executions. +Executions can simply list VMs or also specify benchmark suites. +This gives a lot of flexibility to define the desired combinations. + +**description and desc:** + +Description of the experiment with `description` or `desc`. + +Example: + +```yaml +experiments: + Example: + description: My example experiment. +``` + +--- + +**data_file:** + +The data for this experiment goes into a separate file. +If not given, the `default_data_file` is used. + +Example: + +```yaml +experiments: + Example: + data_file: example.data +``` + +--- + +**reporting:** + +Experiments can define specific reporting options. +See the [reporting](#reporting) for details on the properties. + +Example: + +```yaml +experiments: + Example: + reporting: + codespeed: + ... +``` + +--- + +**suites:** + +List of benchmark suites to be used. + +Example: + +```yaml +experiments: + Example: + suites: + - ExampleSuite +``` + +--- + +**executions:** + +The VMs used for execution, possibly with specific suites assigned. +Thus `executions` takes a list of VM names, possibly with additional keys +to specify a suite and other details. + +Example, simple list of VM names: + +```yaml +experiments: + Example: + executions: + - MyBin1 +``` + +Example, execution with suite: + +```yaml +experiments: + Example: + executions: + - MyBin1: + suites: + - ExampleSuite + cores: [3, 5] +``` + +--- + +**run details and variables:** + +An experiment can additional use the keys for [run details](#runs) and +[variables](#benchmark) (`input_sizes`, `cores`, `variable_values`). +Note, this is possible on the main experiment, but also separately for each +of the defined executions. [merge keys]: http://yaml.org/type/merge.html [node anchors]: http://yaml.org/spec/1.1/current.html#id899912 +[Codespeed]: https://github.com/tobami/codespeed/ diff --git a/docs/extensions.md b/docs/extensions.md index ef9583eb..5433a101 100644 --- a/docs/extensions.md +++ b/docs/extensions.md @@ -8,11 +8,11 @@ and store it in its own data files for later processing. ReBench provides currently builtin support for the following benchmark harnesses: -- [JMH](http://openjdk.java.net/projects/code-tools/jmh/), Java's mircobenchmark harness -- a plain seconds log, i.e., a floating point number per line -- the ReBench log format, which indicates benchmark name and run time in milliseconds or microseconds -- the harness of the [Savina](https://github.com/shamsimam/savina) benchmarks -- a harness that use automatically `/usr/bin/time` +- `JMH`: [JMH](http://openjdk.java.net/projects/code-tools/jmh/), Java's mircobenchmark harness +- `PlainSecondsLog`: a plain seconds log, i.e., a floating point number per line +- `ReBenchLog`: the ReBench log format, which indicates benchmark name and run time in milliseconds or microseconds +- `SavinaLog`: the harness of the [Savina](https://github.com/shamsimam/savina) benchmarks +- `Time`: a harness that use automatically `/usr/bin/time` ## Supporting other Benchmark Harness diff --git a/rebench/rebench-schema.yml b/rebench/rebench-schema.yml index a783e4a3..e247284e 100644 --- a/rebench/rebench-schema.yml +++ b/rebench/rebench-schema.yml @@ -136,9 +136,11 @@ schema;benchmark_suite_type: The command for the benchmark harness. It's going to be combined with the VM's command line. It supports various format variables, including: - benchmark (the benchmark's name) + - cores (the number of cores to be used by the benchmark) - input (the input variable's value) + - iterations (the number of iterations) - variable (another variable's value) - - cores (the number of cores to be used by the benchmark) + - warmup (the number of iterations to be considered warmup) location: type: str desc: | @@ -160,10 +162,10 @@ schema;benchmark_suite_type: - include: benchmark_type_map description: type: str - desc: A description of the benchmark. + desc: A description of the benchmark suite. desc: type: str - desc: A description of the benchmark. + desc: A description of the benchmark suite. schema;vm_type: type: map