Skip to content

Latest commit

 

History

History
121 lines (71 loc) · 5.14 KB

ARCHITECTURE.md

File metadata and controls

121 lines (71 loc) · 5.14 KB

Architecture

This document describes the high-level architecture of Schemathesis.

Core concepts

Schemathesis is initially built around Hypothesis, a framework for property-based testing and inherits a lot of design decisions from it. These are the main steps of how Schemathesis tests API:

  1. Break down an API schema into a set of data generators
  2. Generate test cases with these generators and send them to the API under test
  3. Run a set of checks on the received responses
  4. Report any found failures

Steps 2 to 4 follow the common action - assert pattern and either generated by Schemathesis dynamically, or exposed via its Python API so the users can extend this flow.

However, to run the generators, store results, de-duplicate failures and such, Schemathesis runs this test flow by means provided by Hypothesis. That is, Schemathesis generates a test and provides tools to configure it, then this test is executed by Hypothesis.

It is important to note that Schemathesis has a Python API and integrates with pytest, however, it is main usage scenario is a CLI.

core - low-level core components

These components do not depend on any API specification.

core/schemas

The first layer is how Schemathesis works with API schemas. This package contains various functions to facilitate easy access to different components of API schemas based on a spec.

  • core/schemas/selection. Tools to select a subset of a schema.

core/generator

Another component is data generation where Schemathesis uses Hypothesis & its plugins to produce data samples of needed shape and form. This package contains:

  • core/generator/filters - facilities to detect whether generated samples can be used for a test.

Generally, Schemathesis' generators try to create data without using filters to avoid spending resources on generation, but it is not always easily possible, and filters contain rather performance-sensitive code.

core/errors

Base error handling that is not tied to any specific API specification or execution context. Contains fundamental error types for:

  • Schema processing
  • Data serialization
  • Generic validation

core/marks

A mechanism for attaching Schemathesis-specific metadata to test functions. Since Schemathesis integrates with external testing frameworks (Hypothesis, pytest) that primarily operate on test functions, we need a way to associate additional context with these functions.

core/checks

Tools to check API responses.

core/hooks

A system to extend Schemathesis

core/loaders

Loading API schemas.

core/transports

Utilities for sending & receiving data.

  • core/rate_limit

core/output

Utilities to work with how Schemathesis output its results.

Intermediate-level API

This layer is built on top of core and combines its different components to provide building blocks for concrete API specification. Some parts are re-exported to the public API in order to provide a way to extend Schemathesis.

openapi

  • openapi/checks. Open API-specific checks

graphql

  • graphql/checks. GraphQL-specific checks

engine

Schemathesis Engine brings together other low-level components and executes tests. It is designed to be embeddable and powers Schemathesis CLI. Its execution flow contains separate phases that may share context to improve subsequent phases. The main interface it provides is a stream of events. During execution, the engine collects information about the API and uses it to improve data generation in subsequent phases.

  • engine/phases/probes. Makes requests to learn about API capabilities, for example, whether it accepts NULL bytes or immediately rejects it, so Schemathesis won't spend time generating data with NULL bytes.
  • engine/phases/explicit. Tests based on examples specified in the schema with minimal data generation.
  • engine/phases/coverage. Deterministically generates test cases to cover common "positive" and "negative" testing scenarios depending on the configuration.
  • engine/phases/unit. Unit testing phase. Each available API operation is tested in isolation.
  • engine/phases/stateful. Stateful testing phase. This phase builds connections between different API endpoints and tests them in random sequences reusing response data as input for other operations.

service

Integration with SaaS

High-level API

cli

CLI is built on top of engine and consumes events coming from it. The execution process is based on a set of event handlers that process events. Other than that, handlers use actor-based communication model to share information among other handlers (i.e. ask the output handler to print to stdout or report handler to add an entry to report). This model is intended to decouple handlers, allow graceful error handling, and centralize output tasks (especially for stdout).

  • cli/commands
  • cli/bus
  • cli/validation
  • cli/handlers/junit
  • cli/handlers/vcr
  • cli/handlers/har
  • cli/handlers/debug
  • cli/handlers/stdout

python

A public Python API to core components. Some core components are re-exported here.

  • pytest_plugin. Schemathesis' pytest plugin.
  • asgi. ASGI integration
  • wsgi. WSGI integration