Skip to content

Commit

Permalink
More config settings documented
Browse files Browse the repository at this point in the history
Signed-off-by: Stefan Marr <git@stefan-marr.de>
  • Loading branch information
smarr committed Jul 14, 2018
1 parent b7fbe4d commit 84462c5
Showing 1 changed file with 268 additions and 0 deletions.
268 changes: 268 additions & 0 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,274 @@ runs:
execute_exclusively: false
```

## Reporting

Currently, [Codespeed] is the only supported system for continuous
performance monitoring. It is configured with the `reporting` key.

**codespeed:**

Send results to Codespeed for continuous performance tracking.
The settings define the project that is configured in Codespeed, and the
URL to which the results are reported. Codespeed requires more information,
but since these details depend on the environment, they are passed set on
the [command line](usage.md#continuous-performance-tracking).

Example:

```yaml
reporting:
codespeed:
project: MyVM
url: http://example.org/result/add/json/
```

---

## Benchmark Suites

Benchmark suites are named collections of benchmarks and settings that apply to
all of them.

**gauge_adapter:**

Name of the parser that interpreters the output of the benchmark harness.
For a list of supported options see the list of [extensions](extensions.md#available-harness-support).

This key is mandatory.

Example:

```yaml
benchmark_suites:
ExampleSuite:
gauge_adapter: ReBenchLog
```

---

**command:**

The command for the benchmark harness. It's going to be combined with the
VM's command line. Thus, it should instruct the VM which harness to use
and how to map the various parameters to the corresponding harness settings.

It supports various format variables, including:

- benchmark (the benchmark's name)
- cores (the number of cores to be used by the benchmark)
- input (the input variable's value)
- iterations (the number of iterations)
- variable (another variable's value)
- warmup (the number of iterations to be considered warmup)

This key is mandatory.

Example:

```yaml
benchmark_suites:
ExampleSuite:
command: Harness %(benchmark)s --problem-size=%(input)s --iterations=%(iterations)s
```

---

**location:**

The path to the benchmark harness. Execution use this location as
working directory. It overrides the location/path of a VM.

Example:

```yaml
benchmark_suites:
ExampleSuite:
location: ../benchmarks/
```

---

**build:**

The given string is executed by the system's shell and can be used to
build a benchmark suite. It is executed once before any benchmarks of the suite
are executed. If `location` is set, it is used as working directory.
Otherwise, it is the current working directory of ReBench.

Example:

```yaml
benchmark_suites:
ExampleSuite:
build: ./build-suite.sh
```

---

**description/desc:**

The keys `description` and `desc` can be used to add a simple explanation of
the purpose of the suite.

Example:

```yaml
benchmark_suites:
ExampleSuite:
description: |
This is an example suite for this documentation.
```

---

**benchmarks:**

The `benchmarks` key takes the list of benchmarks. Each benchmark is either a
simple name, or a name with additional properties.
See the section on [benchmark](#benchmark) for details.

Example:

```yaml
benchmark_suites:
ExampleSuite:
benchmark:
- Benchmark1
- Benchmark2:
extra_args: "some additional arguments"
```

---

**run details and variables:**

A benchmark suite can additional use the keys for [run details](#run) and
[variables](#benchmark).
Thus, one can use:

- `invocations`
- `iterations`
- `warmup`
- `min_iteration_time`
- `max_invocation_time`
- `parallel_interference_factor`
- `execute_exclusively`

As well as:

- input_sizes
- cores
- variable_values

## Benchmark

A benchmark can be define simply as a name. However, some times one might want
to define extra properties.

**extra_args:**

This extra argument is appended to the benchmark's command line.

Example:

```yaml
- Benchmark2:
extra_args: "some additional arguments"
```

---

**command:**

ReBench will use this command instead of the name for the command line.

Example:

```yaml
- Benchmark2:
command: some.package.Benchmark2
```

---

**codespeed_name:**

A name used for this benchmark when sending data to Codespeed.
This gives more flexibility to keep Codespeed and these configurations or
source code details decoupled.

Example:

```yaml
- Benchmark2:
codespeed_name: "[peak] Benchmark2"
```

---

**input_sizes:**

Many benchmark harnesses and benchmarks take an input size as a
configuration parameter. It might identify a data file, or some other
way to adjust the amount of computation performed.

`input_sizes` expects a list, either as in the list notation below, or
in form of a sequence literal: `[small, large]`.

Example:

```yaml
- Benchmark2:
input_sizes:
- small
- large
```

---

**cores:**

The number of cores to be used by the benchmark.
At least that's the original motivation for the variable.
In practice, it is more flexible and just another variable that can take
any list of strings.

Example:

```yaml
- Benchmark2:
cores: [1, 3, 4, 19]
```

---

**variable_values:**

Another dimension by which the benchmark execution can be varied.
It takes a list of strings, or arbitrary values really.

Example:

```yaml
- Benchmark2:
variable_values:
- Sequential
- Parallel
- Random
```

---

**run details:**

A benchmark suite can additional use the keys for [run details](#run).

---

## Virtual Machines





Expand Down

0 comments on commit 84462c5

Please sign in to comment.