Test Files

Creating and Using tg_pytest Test Files

General

Master File (cfg file)

• The master file contains all test files to execute in a test run (typically test-master.cfg)
• One file per line
• Lines with files can have comments (#) after the filename
• Lines starting with '#' are not executed
• A line containing "EOF" (all caps) will terminate the file at that point

Test Files (JSON files)

• A JSON test file contains one or more tests (see test-001.json)
• Each test is defined by an independent JSON test object, as described below
• Multiple tests in a file must be separated by at least one '#' comment line
• Comments are Python style. Currently only # is supported, not """
• Open curlies are not allowed in comments '{'
• Blank lines in the file are also OK, but are not test separators

Running a Test

• A test run (invocation of tg_pytest.py) executes all uncommented files in the master file
• Test files run in order, and tests in each file run in order (a single "test")
• A test sends all the "send" strings, collects all the response lines (r's, sr's and er's), then analyzes according to the JSON for that

JSON Files

• Each test must be a valid, parsable JSON object. If in doubt, lint it. It's useful to edit these files in an editor with a built-in JSON linter like Atom (use json-linter package)

"t" object

The "t" object controls the execution of the test. If it is omitted the test will not be run. All other data in the JSON object will be ignored. Recognized tags are:

• "label" is displayed when the test is run. If omitted there will be no label
• "send" is an array of one or more strings to send for the test. If omitted the test label might display, but the test is skipped
• "delay" (future) is an optional delay in seconds between send lines. Values < 1 are OK
• "fail" (future) can be "hard" or "soft" (default if omitted). Hard will abort the test run
• "verbosity" (future) affects the analyzers. It can be one of:
  • 0=silent
  • 1=terse - failures only
  • 2=normal - success and failures
  • 3=verbose - even more

"r" object

The "r" object contains the elements to check in all 'r' responses:

• If "r" is present, test all keys for exact match, e.g. xvm:12000
• Use "status" to match the status in the footer
• Use "count" to match the count in the footer
• Example:

    "r":{"status":0}

"sr" object

The "sr" object contains the elements to check in the status reports. A synthetic status report is constructed that collects the latest value for each key reported (i.el. synthesizing an unfiltered status report). THe "response" string provided is from the last status report. Actions are:

• If "sr" is present, test all keys for exact match
• Example:

    "sr":{"stat":3}

'er' object

The "er" object contains the elements to check in any exception reports

• Exception reports are always displayed unless explicitly disabled using "display"
• If "er" is present, test all keys for exact match (future)
• Use "display" to turn off exception reports. reports will always be displayed unless they are explicitly disabled using display:false

    er:{"display":false}