shakespeare

shakespeare is a scenario-based testing framework for distributed processes.

It orchestrates the execution and monitoring of multiple concurrent test processes and synthetizes test results including plots of activity over time.

• How it works
• Command line parameters
• Example: traffic and red lights
• Example: CockroachDB sustaining client traffic while a node is down
• Reference manual

How it works

1. shakespeare reads a configuration from its standard input that defines:

   • one or more actors, each playing a role (multiple actors can play the same role);
   • a script that defines what the actors should be doing over time;
   • optionally, to obtain diagrams of the behavior:
     • spotlights that shine upon the actors during a play, to extract observable data;
     • an audience of observers during the play;
   • optionally, to verify the behavior:
     • some observers can also be auditors that check whether the actors perform well.

2. shakespeare reads the configuration from its standard input and compiles the script to a sequence of scenes for the actors to follow.

   At each scene, the following can happen:

   • an actor can be called to perform some actions (a command is executed);
   • the mood of the stage can be altered (a period of interest is marked in the output diagrams);

3. shakespeare brings the conductor on stage. The play begins. The conductor does the following:

   • calls the spotlights to shine upon the actors (starts the monitoring processes).
   • invites the prompter to the front of the stage. The prompter instructs the actors to carry out the steps defined by the script (runs action commands).
   • calls the actors to leave the scene (runs the role cleanup commands).

4. meanwhile, each observer in the audience scrutinizes one or more actor as revealed by their spotlight: plottable data is gathered by filtering the monitoring data.

   observers that are also auditors scrutinize the actors and check that their performance is acceptable: behavior signals are checked for specified acceptance criteria defined by predicates.

5. during the play, the prompter reports the scenes and actions on the left side of the terminal output. The observers report collected events on the middle of the terminal output. Audit results, if any, are shown on the right of the terminal output.

6. at the end of the play, the audience reports on their experience (plots and test results are produced). The result plots all share the timeline of the play on the x-axis.

   • the performed actions are displayed in the top plot.
   • each observer then gets its own plot.
   • on each observer plot, each of the observed signal sources (from spotlights) produces a data series, and audit results are overlaid on top.

This overall operation is illustrated by the following diagram:

Example: traffic and red lights

A separate document page introduces the main features of shakespeare using a running example.

The example orchestrates two play characters: a road where traffic occurs, and a traffic light meant to stop or let traffic flow. Example behavior diagram:

The top plot shows the actions carried out by shakespeare on the system. The bottom plot shows how the system reacted.

(link to configuration — link to doc)

Example: CockroachDB sustaining client traffic while a node is down

This example shows the behavior of a 3-node CockroachDB clusters with two "KV workload" clients. The node to which one of the two clients is connected is brought down two times gracefully. The behavior plot shows that the other client's throughput remains stable during the downtime.

The top plot shows the actions carried out by shakespeare on the system. The bottom plots show how the system reacted.

(link to configuration)

Reference manual

A reference manual (currently incomplete) is included in the docs/ sub-directory.