Skip to content

genomoncology/rigor

Repository files navigation

Rigor is a Domain Specific Language (DSL) and Command Line Interface (CLI) for making HTTP requests, extracting data, and validating responses. The main intent of Rigor is to be an HTTP-based API (e.g. REST) Testing Framework for automated functional or integration testing.


Requirements

  • Python 3.11

Installation

Install using pip...

pip install rigor

Feature List

  • Functional testing without the need to write glue code. (e.g. Cucumber)
  • Runs in either synchronous (requests) or asynchronous (aiohttp) mode.
  • YAML-based format for Test Case files for easy test creation and maintenance.
  • Response transformation using jmespath.py to reduce test fragility.
  • Pretty HTML test execution reports using cucumber-sandwich.
  • Swagger path coverage report to ensure API surface area coverage.
  • Syntax highlighted console or JSON-based logging using structlog.
  • Profiles for switching between different environments and settings.
  • Tags and CLI options for selectively executing subsets of the test suite.
  • Scenario Outlines (i.e. Tables) for cases with numerous scenarios.
  • Beautiful Soup parsing for extraction from HTML data.
  • Proper error code ($?) on suite success (0) or failure (!0)
  • Case-scenario unique identifier (uuid) for managing session and race conditions.

Command Line Interface (CLI) Options

$ rigor --help
Usage: rigor [OPTIONS] [PATHS]...

Options:
  --profile TEXT             Profile name (e.g. test)
  --host TEXT                Host name (e.g. http://localhost:8000)
  -i, --includes TEXT        Include tag of cases. (e.g. smoke)
  -e, --excludes TEXT        Exclude tag of cases to run. (e.g. broken)
  -p, --prefixes TEXT        Filter cases by file prefix. (e.g. smoke_)
  -e, --extensions TEXT      Filter cases by file extension. (e.g. rigor)
  -c, --concurrency INTEGER  # of concurrent HTTP requests. (default: 5)
  -o, --output TEXT          Report output folder.
  -q, --quiet                Run in quiet mode. (warning/critical level only)
  -v, --verbose              Run in verbose mode. (debug level logging)
  -j, --json                 JSON-style logging.
  -h, --html                 Generate HTML report.
  -g, --coverage             Generate Coverage report.
  -r, --retries INTEGER      # of retries for GET calls only. (default: 0)
  -s, --sleep INTEGER        Retry sleep (seconds multiplied by retry).
                            (default: 60)
  -f, --retry_failed         Retries all failed scenarios at the end.
  --version                  Logs current version and exits.
  --help                     Show this message and exit.

Simple Example

(rigor) /p/tmp> cat test.rigor
name: Simple case.

steps:
  - description: Simple step.
    request:
      host: https://httpbin.org
      path: get

(rigor) /p/tmp> rigor test.rigor --html
2018-02-08 13:18.06 [info     ] no config file not found       [rigor] paths=('test.rigor',)
2018-02-08 13:18.06 [info     ] collecting tests               [rigor] cwd=/private/tmp paths=['test.rigor']
2018-02-08 13:18.06 [info     ] tests collected                [rigor] queued=1 skipped=0
2018-02-08 13:18.06 [info     ] execute suite complete         [rigor] failed=0 passed=1 timer=0.119s
2018-02-08 13:18.07 [info     ] launching browser              [rigor] report_path=/var/folders/b_/2hlrn_7930x81r009mfzl50m0000gn/T/tmps_8d7nn_/html-2018-02-08-08-18-06/cucumber-html-reports/cucumber-html-reports/overview-features.html

list

detail

Object Model

  • suite: set of cases that gets built dynamically based on cli arguments.
  • case: set of scenarios and steps in a .rigor file.
  • scenario: namespace for 1 run of case steps.
  • step: request with response extract, validate, etc.
  • iterate: repeats an individual step by iterating through iterable.
  • request: http call (get, post, etc.) to path with parameters, data, or uploads
  • extract: extract nested data from a response into a variable available to following steps.
  • validate: check an actual value against an expected value using a comparator.
  • transform: using jmespath to shape a JSON response into a specific format.

objects

Comparators

Comparators are used by the validation phase of each step to check whether an actual value is returning as expected. See the comparisons.rigor example for more details.

  • equals
  • not equals
  • same
  • not same
  • greater than
  • less than
  • greater than or equals
  • less than or equals
  • type
  • in
  • not in
  • regex
  • subset
  • not subset
  • length
  • superset
  • not superset
  • keyset
  • not keyset
  • contains
  • not contains

Related Projects

  • Tavern is an extremely similar project that was released a little too late for us to use.
  • Pyresttest was the first library we used before deciding to roll our own testing framework.
  • Click is the library used to build out the command-line options.
  • Related is the library used for parsing the YAML test suite into an Python object model.

More Examples

More examples can be found by reviewing the tests/httpbin/ folder of this project.

License

The MIT License (MIT) Copyright (c) 2017 Ian Maurer, Genomoncology LLC