DACBench: A Benchmark Library for Dynamic Algorithm Configuration

DACBench is a benchmark library for Dynamic Algorithm Configuration. Its focus is on reproducibility and comparability of different DAC methods as well as easy analysis of the optimization process.

You can try out the basics of DACBench in Colab here without any installation. Our examples in the repository should give you an impression of what you can do with DACBench and our documentation should answer any questions you might have.

Installation

We recommend installing DACBench in a virtual environment, here with uv:

pip install uv
uv venv --python 3.10
source .venv/bin/activate
uv pip install dacbench

Instead of using pip, you can also use the GitHub repo directly:

pip install uv
git clone https://github.com/automl/DACBench.git
cd DACBench
uv venv --python 3.10
source .venv/bin/activate
git submodule update --init --recursive
make install

This command installs the base version of DACBench including the three small surrogate benchmarks. For any other benchmark, you may use a singularity container as provided by us (see next section) or install it as an additional dependency. As an example, to install the SGDBenchmark, run:

uv pip install dacbench[sgd]

You can also install all dependencies like so:

make install-dev

Containerized Benchmarks

DACBench can run containerized versions of Benchmarks using Singularity containers to isolate their dependencies and make reproducible Singularity images.

Building a Container

For writing your own recipe to build a Container, you can refer to dacbench/container/singularity_recipes/recipe_template

Install Singularity and run the following to build the (in this case) cma container

cd dacbench/container/singularity_recipes
sudo singularity build cma cma.def

Citing DACBench

If you use DACBench in your research or application, please cite us:

@inproceedings{eimer-ijcai21,
  author    = {T. Eimer and A. Biedenkapp and M. Reimer and S. Adriaensen and F. Hutter and M. Lindauer},
  title     = {DACBench: A Benchmark Library for Dynamic Algorithm Configuration},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence ({IJCAI}'21)},
  year      = {2021},
  month     = aug,
  publisher = {ijcai.org},