Skip to content

Latest commit

 

History

History
261 lines (215 loc) · 10.3 KB

BatchEvaluation.md

File metadata and controls

261 lines (215 loc) · 10.3 KB
Raw

Batch Evaluation of Square Root Optimization and Marginalization

In this tutorial we detail how you can use the batch evaluation scripts to reproduce the results of the ICCV'21 paper Demmel et al., "Square Root Marginalization for Sliding-Window Bundle Adjustment". In the paper we discuss how square root estimation techniques can be used in Basalt's optimization-based sliding-window odometry to make optimization faster and marginalization numerially more stable. See the project page for further details.

Basalt's VIO/VO now runs with single-precision floating point numbers by default, using the new square root formulation. The conventional squared (Hessian-based) formualtion is still available via config options. For manual testing, you can pass --use-double true or --use-double false (default) as command line arguments to basalt_vio, and change config.vio_sqrt_marg in the config file, which controls if the marginalization prior is stored in Hessian or Jacobian form (default: true), as well as config.vio_linearization_type in the config file, which controls whether to use Schur complement, or nullspace projection and QR-decomposition for optimization and marginalization ("ABS_SC" or "ABS_QR" (default)).

In the following tutorial we systematically compare the different formulations in single and double precision to reproduce the results from the ICCV'21 paper. You can of course adjust the correspondig config files to evaluate other aspects of the system.

Prerequisites

  1. Source installation of Basalt: The batch evaluation scripts by default assume that the build folder is directly inside the source checkout basalt. See README.md for instructions.
  2. Downloads of the datasets: We evaluate EuRoC (all 11 sequneces), TUMVI (euroc format in 512x512 resultion; sequences: corridor1-2, magistrale1-2, room1-2, slides1-2), and Kitti Odometry (sequences 00-10). It's recommended to store the data locally on an SSD to ensure that reading the images is not the bottleneck during evaluation (on a multicore desktop Basalt runs many times faster than real-time). There are instructions for downloading these dataset: EuRoC, TUMVI, KITTI. Calibration for EuRoC and TUMVI is provided in the data folder. For KITTI you can use the basalt_convert_kitti_calib.py script to convert the provided calibration to a Basalt-compatible format (see KITTI).
  3. Dependencies of evaluation scripts: You need pip packages py_ubjson, matplotlib, numpy, munch, scipy, pylatex, toml. How to install depends on your Python setup (virtualenv, conda, ...). To just install for the local user with pip you can use the command python3 -m pip install --user -U py_ubjson matplotlib numpy munch scipy pylatex toml. For generating result tables and plots you additionally need latexmk and a LaTeX distribution (Ubuntu: sudo apt install texlive-latex-extra latexmk; macOS with Homebrew: brew install --cask mactex).

Folder Structure

The batch evaluation scripts and config files assume a certain folder structure inside a "parent" folder, since relative paths are used to find the compiled executable and calibration files. So it's important to follow the folder structure.

parent-folder/
├─ basalt/
│  ├─ build/
│  │  ├─ basalt_vio
│  │  ├─ ...
│  ├─ data/
│  │  ├─ euroc_ds_calib.json
│  │  ├─ ...
│  ├─ ...
├─ experiments/
│  ├─ iccv_tutorial/
│  │  ├─ basalt_batch_config.toml
│  │  ├─ experiments-iccv.toml
│  │  ├─ 01_iccv_all/
│  │  │  ├─ ...
│  │  ├─ 02_iccv_runtime/
│  │  │  ├─ ...

As a sibling of the basalt source checkout we'll have an experiments folder, and inside, a folder iccv_tutorial for this tutorial. Into that folder, we copy the provided basalt_batch_config.toml file that defines the configurations we want to evaluate and from which we generate individual config files for each VIO / VO run. We also copy the provided experiments-iccv.toml config file, which defines the results tables and plots that we generate from the experiments' logs.

Note: Commands in this tutorial are assumed to be executed from within parent-folder unless specified otherwise.

mkdir -p experiments/iccv_tutorial
cp basalt/data/iccv21/basalt_batch_config.toml experiments/iccv_tutorial/
cp basalt/data/iccv21/experiments-iccv.toml experiments/iccv_tutorial/

Generate Experimental Configs

First, edit the copied configuration file experiments/iccv_tutorial/basalt_batch_config.toml and modify all "dataset-path" lines to point to the locations where you downloaded the datasets to.

Now, we can generate per-experiment config files:

cd experiments/iccv_tutorial/
../../basalt/scripts/batch/generate-batch-configs.py .

This will create subfolder 01_iccv_all containing folders vio_euroc, vio_tumvi, and vo_kitti, which in turn contain all generated basalt_config_...json files, one for each experiment we will run.

Run Experiments

We can now run all experiments for those generate configs. Each config / sequence combination will automatically be run twice and only the second run is evaluated, which is meant to ensure that file system caches are hot.

Since we also evaluate runtimes, we recommend that you don't use the machine running the experiments for anything else and also ensure no expensive background tasks are running during the evaluation. On one of our desktops with 12 (virtual) cores the total evaluation of all sequences takes aroudn 3:30h. Your milage may vary of course depending on the hardware.

cd experiments/iccv_tutorial/
time ../../basalt/scripts/batch/run-all-in.sh 01_iccv_all/

Inside 01_iccv_all, a new folder with the start-time of the experimental run is created, e.g., 20211006-143137, and inside that you can again see the same per-dataset subfolders vio_euroc, vio_tumvi, and vo_kitti, inside of which there is a folder for each config / run. Inside these per-run folders you can find log files including the command line output, which you can inspect in case something doesn't work.

In a second terminal, you can check the status of evaluation while it is running (adjust the argument to match the actual folder name).

cd experiments/iccv_tutorial/
../../basalt/scripts/batch/list-jobs.sh 01_iccv_all/20211006-143137

If you see failed experiments for the square root solver in single precision, don't worry, that is expected.

Generate Results Tables and Plots

After all experimental runs have completed, you can generate a PDF file with tabulated results and plots, similar to those in the ICCV'21 paper.

cd experiments/iccv_tutorial/
../../basalt/scripts/batch/generate-tables.py --config experiments-iccv.toml --open

The results are in the generated tables/experiments-iccv.pdf file (and with the --open argument should automatically open with the default PDF reader).

Better Runtime Evaluation

The experiments above have the extended logging of eigenvalue and nullspace information enabled, which does cost a little extra runtime. To get a better runtime comparison, you can re-run the experiments without this extended logging. The downside is, that you can only generate the results tables, but not the plots.

We assume that you have already followed the tutorial above, including the initial folder setup. For these modified experiments, we redo all three steps (generating config files; running experiments; generating results) with slight modifications.

First, edit the experiments/iccv_tutorial/basalt_batch_config.toml file at the bottom, and uncomment the commented entries in _batch.combinations as well as the commented revision. At the same time, comment out the initially uncommented lines. It should look something like this after the modifications:

[_batch.combinations]
#vio_euroc = ["vio",             "savetumgt", "extlog", "runtwice", "all_meth", "all_double", "all_euroc"]
#vio_tumvi = ["vio", "tumvivio", "savetumgt", "extlog", "runtwice", "all_meth", "all_double", "more_tumvi"]
#vo_kitti  = ["vo",  "kittivo",  "savetumgt", "extlog", "runtwice", "all_meth", "all_double", "all_kitti"]

vio_euroc = ["vio",             "runtwice", "all_meth", "all_double", "all_euroc"]
vio_tumvi = ["vio", "tumvivio", "runtwice", "all_meth", "all_double", "more_tumvi"]
vo_kitti  = ["vo",  "kittivo",  "runtwice", "all_meth", "all_double", "all_kitti"]
[_batch]
#revision = "01_iccv_all"
revision = "02_iccv_runtime"

You can see that we removed the savetumgt and extlog named config elements and that generated config files and results for this second run of experiments will be placed in 02_iccv_runtime.

Now generate config files and start the experimental runs:

cd experiments/iccv_tutorial/
../../basalt/scripts/batch/generate-batch-configs.py .
time ../../basalt/scripts/batch/run-all-in.sh 02_iccv_runtime/

Before generating the results PDF you need to now edit the experiments-iccv.toml file, point it to the new location for experimental logs and disable the generation of plots. Check the place towards the start of the file where substitutions for EXP_PATTERN_VIO and EXP_PATTERN_VO are defined, as well as SHOW_TRAJECTORY_PLOTS, SHOW_EIGENVALUE_PLOTS, and SHOW_NULLSPACE_PLOTS. After your modifications, that section should look something like:

###################
## where to find experimental runs
[[substitutions]]

#EXP_PATTERN_VIO = "01_iccv_all/*-*/vio_*/"
#EXP_PATTERN_VO = "01_iccv_all/*-*/vo_*/"

EXP_PATTERN_VIO = "02_iccv_runtime/*-*/vio_*/"
EXP_PATTERN_VO = "02_iccv_runtime/*-*/vo_*/"


###################
## which kind of plots to show
[[substitutions]]
SHOW_TRAJECTORY_PLOTS = false
SHOW_EIGENVALUE_PLOTS = false
SHOW_NULLSPACE_PLOTS = false

Now we can generate the results tables for the new experimental runs with the same command as before:

cd experiments/iccv_tutorial/
../../basalt/scripts/batch/generate-tables.py --config experiments-iccv.toml --open