RandomQueryGeneratorCombinations

Principle

Certain bugs will only show up under certain workloads. Testing all possible workloads systematically is computationally impossible, however a probabilistic approach can cover a considerable portion of the search space.

To be able to use RQG's combinations script (combinations.pl) effectively, one first has to understand grammar masking

Combinations

The Random Query Generator provides a facility to run different tests based on a set of common startup options and a common grammar. The options are described in a configuration file and each test run will pick a random configuration and run the desired grammar using a random --mask parameter.

The configuration file has the following structure:

 $combinations = [
        [
        '
                --engine=InnoDB
                --reporters=Deadlock,ErrorLog,Backtrace,Recovery,Shutdown
        '
        ],
        [
                '--mysqld=--transaction-isolation=READ-COMMITTED',
                '--mysqld=--transaction-isolation=REPEATABLE-READ',
        ],
        [
                '--rows=10',
                '--rows=10000'
        ],
        [
                '--threads=4',
                '--threads=64'
        ],

Each section (between [ and ]) lists one or more blocks (a block starts and ends with '). Each block can contain one or more RQG options.

For each run, one block from each section will be picked randomly (based on the --seed value). The options in these selected blocks are then combined to form runall.pl command lines which combinations.pl sequentially executes.

When any section lists just a single block with no alternatives, then the options in that block will be present for each test run.

For added clarity, it makes sense to include only a single block in the first section of the configuration file, and to lists any default RQG options (to be used for each test execution) in this block.

To run a test suite based on the configuration file:

 perl combinations.pl \
   --basedir=/path/to/mysql/basedir \
   --vardir=/path/to/vardir \
   --config=conf/example.cc \
   --grammar=conf/example.yy \
   --trials=100 \
   --duration=1200

This will start 100 tests, 20 min each, with each test picking a random combination of startup options from the configuration file. In addition, a random --mask value will also be generated.

Warning: --mask is applied automatically by RQG as per the above. However, please note that applying --mask only makes sense for DML, concurrent stress workloads, e.g. For example, in a DELETE, UPDATE and INSERT grammar, a random --mask will create an update-only, insert-only, etc. workload from a single grammar. However, if you are only testing single-threaded SELECT statements, then applying a mask value essentially causes some potential queries to not be generated at all, which is not good as it reduces test coverage. In this case, there is an option --no-mask which you can use. This option prevents the automatic generation of a mask value from happening.

The logs from the individual tests will be placed in --vardir as trial###.log files. In addition, the vardir from the current test ("current"), and every failing test (vardir###), will also be preserved.

The grammar to use can be specified either using the --grammar option to combinations.pl, or by placing a --grammar inside the .cc file.

Category:RandomQueryGenerator