Regression Testing

How do we run regression tests?

Regression tests are run via a python script in the RegressionTest folder. It loads up the command line definitions for a bunch of depthmapXcli runs from the regressiontestconfig.json file, and then runs these command lines with a baseline binary and the binary it finds in the build folder, comparing the outputs.

Building the test binary

If it doesn't exist, create a directory called build in the main directory of your repository. Change into it, and run a build from the command line (using qmake and make/nmake). This should create the test binary in the correct place.

Running the tests

Change into the RegressionTest folder and run RegressionTestRunner.py. This requires python >= 3.5. This will create run directories (in the rundir directory) named after the tests, postfixed with _base and _test. Any files that are created by the runs, and the console output will be left in these directories after the run finishes.
A new run will clear any existing directories, so make sure you stow away any files you want to compare across runs before rerunning.

Changing tests

Modifying a test

If you think a test needs to be modified to cover what it is supposed to test, you can change the command line in regressiontestconfig.json file. The modified command line will automatically apply to the test and the baseline run. You need to make sure that it is still compatible with the baseline version after changing. If it isn't, you need to update the baseline executable first.

Adding a test

Add a new test with a new name in regressiontesconfig.json. Have a look at the structure of other tests... Make sure the name can be used in a directory name, as this will happen to create the run directory.

Simple case

If the test is compatible with the current RegressionTestRunner and the baseline binary, just adding the test should be good enough.

Changes to the RegressionTestRunner

If changes to the regression test runner are required (e.g. different command line/config options, different file diffs etc.), it is probably a good idea to make the changes to the RegressionRunner first and add the test after the new version of the regression runner still passes the tests. Please note that the RegressionRunner has its own set of unit tests that need to be updated as it is changed. We aim to cover 100% of the RegressionRunner in unit tests.

New baseline executable

Sometimes, it is necessary to update the base line executable (e.g. if a new command line parameter has been added to depthmapXcli). In this case the procedure should be:

• Create the new version of the code, push it to github and do pull requests etc.
• Make sure it passes all existing regression tests (goes green in travis)
• Then make a release build for Windows, MacOs X and linux from the same version pulled off github, and replace the baseline binaries with these files. To create a linux build, you can use the docker image blackseamonster/depthmapx-buildenv if you do not have a linux machine.
• Make sure the tests go green again
• Only after that can any changes to tests be implemented that require these changes