OpenActive Test Suite

To join the conversation, we're on the OpenActive Slack at #openactive-test-suite.

The general aim of this project is to allow end to end testing of the various flows and failure states of the Open Booking API.

Running npm start in the root will run the OpenActive Test Suite, which is actually comprised of these packages:

• Integration Tests: this performs automated tests against the API.
• Broker Microservice: this sits in between the test suite and the target Open Booking API implementation. This allows the integration tests to watch for changes to the various RPDE feeds.
• OpenID Client, OpenID Browser Automation and OpenID Test CLI: together these connect to the target Open Booking API's OpenID Provider. This allows the Broker and Integration tests to authorize with the implementation
• Test Interface Criteria: this allows test suite to tailor specific opportunities to specific tests by implementing the OpenActive Test Interface Criteria.

Usage

Running npm start will orchestrate running the Broker Microservice and the Integration Tests in order to test your Open Booking API implementation.

Note that the implementation under test will need to implement the OpenActive Test Interface to run in controlled mode, and for selected tests.

Quick start

You can check that the test suite works in your local environment by running it against the hosted OpenActive Reference Implementation, simply by using the default configuration:

git clone git@github.com:openactive/openactive-test-suite.git
cd openactive-test-suite
nvm use
npm install
npm start -- core

Note that the above command only runs the "core" tests within the test suite, which should take around 60 seconds to complete.

The hosted OpenActive Reference Implementation is running on a basic developer tier Azure instance with a burst quota, so it will not handle the load of a test suite run for all tests (hence npm start -- core); if the hosted application shuts down, simply wait 5 minutes and try again.

Configuration

In order to run the test suite against your own implementation, configure the test suite by creating a copy of config/default.json named config/{NODE_ENV}.json (where {NODE_ENV} is the value of your NODE_ENV environment variable), including the following properties:

• broker microservice configuration
• integrationTests and sellers configuration

The test suite uses the file config/{NODE_ENV}.json to override the settings in default.json. For development and deployment create a new file instead of making changes to the default.json file, so that any new required settings that are added in future versions can be automatically updated in default.json.

For more information about this use of NODE_ENV see this documentation.

By convention, much of the documentation assumes you to have created a config/dev.json file, which Test Suite will use when env var NODE_ENV=dev. But you can use any name you like, and have multiple configuration files for different environments.

Configuration for sellers within ./config/{NODE_ENV}

The primary Seller is used for all tests, and random opportunities used when "useRandomOpportunities": true are selected from this Seller. The secondary Seller is used only for multiple-sellers tests.

An example, using OpenID Connect Authentication:

  "sellers": {
    "primary": {
      "@type": "Organization",
      "@id": "https://reference-implementation.openactive.io/api/identifiers/sellers/0",
      "authentication": {
        "loginCredentials": {
          "username": "test1",
          "password": "test1"
        }
      },
      "taxMode": "https://openactive.io/TaxGross",
      "paymentReconciliationDetails": {
        "name": "AcmeBroker Points",
        "accountId": "SN1593",
        "paymentProviderId": "STRIPE"
      }
    },
    "secondary": {
      "@type": "Person",
      "@id": "https://reference-implementation.openactive.io/api/identifiers/sellers/1",
      "authentication": {
        "loginCredentials": {
          "username": "test2",
          "password": "test2"
        }
      },
      "taxMode": "https://openactive.io/TaxNet"
    }
  }

Description of each field:

• authentication: Check out the Configuration for Seller Authentication section.

• taxMode: Which Tax Mode is used for this Seller.

  Note: If testing both Tax Modes, make sure that there is at least one Seller with each. Alternatively, if not supporting multiple Sellers, you can run the Test Suite once with "taxMode": "https://openactive.io/TaxNet" and once with "taxMode": "https://openactive.io/TaxGross". However, it is not currently possible to generate a certificate that covers both configurations unless multiple Sellers are supported.

• paymentReconciliationDetails: If testing Payment Reconciliation Detail Validation, include the required payment reconciliation details here.

Configuration for Seller Authentication

In order to make bookings for a specific Seller's Opportunity data, some kind of authentication is required to ensure that the caller is authorized to make bookings for that Seller.

Test Suite allows for a few different options for Seller Authentication. This determines the data to put in the authentication field for each Seller:

OpenID Connect

View Spec

You'll need the username/password that the Seller can use to log in to your OpenID Connect Provider.

Example:

  "sellers": {
    "primary": {
      // ...
      "authentication": {
        "loginCredentials": {
          "username": "test1",
          "password": "test1"
        }
      }
    },

In order for Test Suite to be able to log in to your OpenID Connect Provider, you'll need to also configure broker.loginPagesSelectors (see Broker Microservice Configuration).

Request Headers

Just a set of request HTTP headers which will be used to make booking requests. There are no restrictions on the requestHeaders that can be specified.

Example:

  "sellers": {
    "primary": {
      // ...
      "authentication": {
        "loginCredentials": null,
        "requestHeaders": {
          "X-OpenActive-Test-Client-Id": "booking-partner-1",
          "X-OpenActive-Test-Seller-Id": "https://localhost:5001/api/identifiers/sellers/1"
        }
      }
    },

Client Credentials

OAuth Client Credentials are used to make booking requests.

Example:

  "sellers": {
    "primary": {
      // ...
      "authentication": {
        "loginCredentials": null,
        "clientCredentials": {
          "clientId": "clientid_XXX",
          "clientSecret": "example"
        }
      }
    },

This is different from the behaviour in the Client Credentials sub-section mentioned within the OpenID Connect Booking Partner Authentication for Multiple Seller Systems section in the spec as, in this case, Client Credentials are used to make booking requests for this Seller, rather than just to view the Booking Partner's Orders Feed.

Installation

First, ensure that you have the correct Node.js version installed. We recommend managing this with Node Version Manager (NVM). If you use NVM, switch to the correct version of Node.js with:

nvm use

Otherwise, get the correct version of Node.js from the .nvmrc file.

Then, install the dependencies needed for all packages in the test suite:

npm install

For developers that are customising the installation, for use in e.g. Docker, the directories ./packages/openactive-openid-browser-automation, ./packages/openactive-openid-client, ./packages/openactive-openid-test-cli, ./packages/openactive-openid-test-client and ./packages/test-interface-criteria are dependencies, and so must be present during npm install.

Running

Where dev.json is the name of your {NODE_ENV}.json configuration file:

export NODE_ENV=dev
npm start

This will start the broker microservice (openactive-broker-microservice) and run all integration tests (openactive-integration-tests) according to the feature configuration. It will then kill the broker microservice upon test completion. The console output includes both openactive-broker-microservice and openactive-integration-tests. This is perfect for CI, or simple test runs.

Alternatively the Broker microservice and Integration tests may be run separately, for example in two different console windows. This is more useful for debugging.

Running specific tests

Any extra command line arguments will be passed to jest in openactive-integration-tests. For example:

export NODE_ENV=dev
npm start -- --runInBand test/features/core/availability-check/

It is also possible to use a category identifier or feature identifier as short-hand:

export NODE_ENV=dev
npm start -- core

export NODE_ENV=dev
npm start -- availability-check

Read about Jest's command line arguments in their CLI docs.

Running with Docker

Alternatively, Docker can be used to run Test Suite.

An example command to run Test Suite, with Docker, for the core tests, with the configuration in ./config/dev.json and output in ./output/ is as follows:

docker run -it \
  -e INPUT_CONFIG=/config/dev.json \
  -e NODE_ENV=dev \
  -p 3000:3000 \
  -v ${PWD}/config:/config \
  -v ${PWD}/packages/openactive-integration-tests/output:/openactive-test-suite/packages/openactive-integration-tests/output \
  ghcr.io/openactive/test-suite:latest \
  -- core

This is equivalent to running Test Suite with NODE_ENV=dev npm start -- core.

An explanation of each part of this command:

• Use -it to get the full interactive Test Suite experience.

  docker run -it \

• Copy the dev.json config file into the Docker container and instruct Test Suite to use it. Usually Test Suite is either configured using config/dev.json or via NODE_CONFIG.

    -e INPUT_CONFIG=/config/dev.json` \
    -e NODE_ENV=dev \

• Expose Broker Microservice's HTTP server to port 3000. This can be useful as the Broker Microservice API can help debug issues.

    -p 3000:3000 \

• Allow the Docker container to read the config file from ./config and to output results to ./packages/openactive-integration-tests/output (assuming that you have the default value, ./output/ configured for integrationTests.outputPath in your dev.json config file)

    -v ${PWD}/config:/config \
    -v ${PWD}/packages/openactive-integration-tests/output:/openactive-test-suite/packages/openactive-integration-tests/output \

• Use the latest Test Suite docker image

    ghcr.io/openactive/test-suite:latest \

• Run all the tests from the core category (see Running specific tests for more information)

    -- core

Further reading:

• Running with Docker for Continuous Integration
• See the documentation within Dockerfile to see how it is built.

Running from a local Docker image

1. Build the image

   docker build -t local-openactive-test-suite

2. Run the image

   docker run -it <OPTIONS> local-openactive-test-suite <...ARGS>

   See Running with Docker for more information on the options and arguments.

Environment variables

NODE_CONFIG

The configuration of the test suite can be overridden with the environment variable NODE_CONFIG, where any specified configuration will override values in both config/default.json. More detail can be found in the node-config docs. For example:

NODE_CONFIG='{ "waitForHarvestCompletion": true, "datasetSiteUrl": "https://localhost:5001/openactive", "sellers": { "primary": { "@type": "Organization", "@id": "https://localhost:5001/api/identifiers/sellers/0", "requestHeaders": { "X-OpenActive-Test-Client-Id": "test", "X-OpenActive-Test-Seller-Id": "https://localhost:5001/api/identifiers/sellers/0" } }, "secondary": { "@type": "Person", "@id": "https://localhost:5001/api/identifiers/sellers/1" } }, "useRandomOpportunities": true, "generateConformanceCertificate": true, "conformanceCertificateId": "https://openactive.io/openactive-test-suite/example-output/random/certification/" }' npm start

PORT

Defaults to 3000.

Set PORT to override the default port that the openactive-broker-microservice will expose endpoints on for the openactive-integration-tests. This is useful in the case that you already have a service using port 3000.

FORCE_COLOR

E.g. FORCE_COLOR=1

Set this to force the OpenActive Test Suite to output in colour. The OpenActive Test Suite uses chalk, which attempts to auto-detect the color support of the terminal. For CI environments this detection is often inaccurate, and FORCE_COLOR=1 should be set manually.

Continuous Integration

Assuming configuration is set using the NODE_CONFIG environment variable as described above, the test suite can be run within a continuous integration environment, as shown below:

#!/bin/bash
set -e # exit with nonzero exit code if anything fails

# Get the latest OpenActive Test Suite
git clone git@github.com:openactive/openactive-test-suite.git
cd openactive-test-suite

# Install dependencies
npm install

# Start broker microservice and run tests
npm start

"ci": true must be included in the supplied NODE_CONFIG to ensure correct console logging output within a CI environment.

Note that running npm start in the root openactive-test-suite directory will override waitForHarvestCompletion to true in default.json, so that the openactive-integration-tests will wait for the openactive-broker-microservice to be ready before it begins the test run.

CI - Docker

You may find it useful to run Test Suite with Docker in your CI environment (general information on running with Docker here).

Similarly to the Continuous Integration section, if we assume that configuration is set using the NODE_CONFIG environment variable, CI can use a command like the following:

docker run -it \
  -e NODE_CONFIG=< PUT YOUR NODE CONFIG HERE > \
  -p 3000:3000 \
  -v ${PWD}/config:/config \
  # Change this if using a custom `integrationTests.outputPath`
  -v ${PWD}/packages/openactive-integration-tests/output:/openactive-test-suite/packages/openactive-integration-tests/output \
  ghcr.io/openactive/test-suite:latest

And as in the parent section, NODE_CONFIG must include "ci": true.

Some things to note:

• When using a hostname other than localhost for your booking system, it must included at least one . to pass the validator.
• host.docker.internal must be the host to access your booking system locally if it is not running in another Docker container. This hostname must also be used within NODE_CONFIG.

CI - GitHub Actions

This repository can also be referenced as a GitHub action, which conveniently wraps the Docker container.

The following steps may be used within a GitHub Actions script:

    steps:
      - name: Create output dirs
        run: mkdir -p ./test-suite/output/json ./test-suite/conformance/
      - name: Run OpenActive Test Suite
        uses: openactive/openactive-test-suite@master
        with:
          config_file: ./test-suite-config.json
          NODE_CONFIG: |
            {"ci": true, "broker": {"outputPath": "/github/workspace/test-suite/output/", "datasetSiteUrl": "http://host.docker.internal/openactive"}, "integrationTests": { "outputPath": "/github/workspace/test-suite/output/", "conformanceCertificatePath": "/github/workspace/test-suite/conformance/", "conformanceCertificateId": "https://certificates.example.com/openactive/" }, "sellers": {"primary": {  "@id": "http://host.docker.internal/api/identifiers/sellers/1","secondary": {  "@id": "http://host.docker.internal/api/identifiers/sellers/2"}}}}
      - name: Upload test output as artifact
        uses: actions/upload-artifact@v4
        if: ${{ success() || failure() }}
        with:
          name: openactive-test-suite
          path: ./test-suite/output/
      - name: Deploy conformance certificate to Azure Blob Storage (master branch only)
        uses: bacongobbler/azure-blob-storage-upload@v1.2.0
        if: ${{ github.ref == 'refs/heads/master' }}
        with:
          source_dir: ./test-suite/conformance/
          container_name: '$web'
          connection_string: ${{ secrets.CONFORMANCE_CERTIFICATE_BLOB_STORAGE_CONNECTION_STRING }}
          sync: false

And as in the parent section, NODE_CONFIG must include "ci": true.

Note that outputPath and conformanceCertificatePath must start with /github/workspace/ to ensure these outputs are accessible in subsequent steps.

Sub-directories must be created before the test suite runs (including a subdirectory of the output folder named json) e.g. mkdir -p ./test-suite/output/json ./test-suite/conformance/.

As in the previous section, host.docker.internal must be the host to access your booking system locally to the GitHub action if it is not running in another Docker container. This hostname must also be used within NODE_CONFIG for datasetSiteUrl and Seller @ids.

The Test Suite Certificate should be updated upon each successfull CI run.

Test Data Requirements

In order to run the tests in random mode, the target Open Booking API implementation will need to have some Opportunity data pre-loaded. Use Test Data Generator to find out how much data is needed and in what configuration.

Certification

An OpenActive Conformance Certificate offers a mechanism by which implementing systems can prove their conformance to the OpenActive specifications. Test Suite can be configured to output a Conformance Certificate upon all tests passing.

An example conformance certificate can be found here: https://certificates.reference-implementation.openactive.io/examples/all-features/random/

For more information about Certification please see here.

Contributing

• Contributing to the project

Concepts

Booking Partner Authentication Strategy

The method by which a Booking Partner authenticates with the Open Booking API implementation. There are a number of supported strategies, including OpenID Connect, HTTP Header, etc.

Your impementation will need to support at least one Authentication Strategy for each of Orders Feed Authentication and Booking Authentication.

Orders Feed Authentication

How a Booking Partner accesses the Orders Feed containing updates to Orders that they have created.

For Test Suite, the selected Orders Feed Authentication Strategy is configured with the broker.bookingPartners configuration property and documentation on the supported strategies can be found there.

Booking Authentication

How a Booking Partner accesses the booking endpoints (C1, C2, B, etc) for a specific Seller's data. This differs from Orders Feed Authentication as it can be specified at the per-Seller level for Multiple Seller Systems (relevant feature: multiple-sellers).

For Test Suite, the selected Booking Authentication Strategy is configured with the sellers configuration property and documentation on the supported strategies can be found there.