CMIP Rapid Evaluation Framework

Status: This project is in active development. We expect to be ready for beta releases in Q2 2025.

The CMIP Rapid Evaluation Framework is a Python application that provides the ability to rapidly process and evaluate CMIP data against a set of reference data. It is designed to be used as a CI/CD pipeline to provide a quick validation of CMIP data.

CMIP REF is a community project, and we welcome contributions from anyone.

PyPI :

Other info :

Getting started

As a metrics provider

Metrics providers are the core of the REF. They define the metrics that will be calculated and the data that will be used to calculate them, by providing a consistent interface for the REF to interact with.

These metrics providers can be run as standalone applications or as part of the REF. See

As a modelling center

The REF is designed to enable Modelling Centers to quickly evaluate their data against a set of reference data. The data under test here may not be published to ESGF yet, but the REF can still be used to evaluate it.

TODO: Docs for that workflow

ref datasets ingest {data_path} --solve

As a devops engineer

The REF can also be deployed as a standalone set of services that don't require any user interaction. This is useful for running the REF to automatically evaluate data as it is published to ESGF.

TODO: Docs for that workflow

Each service in the REF is designed to be run as a separate docker container.

Full documentation can be found at: cmip-ref.readthedocs.io. We recommend reading the docs there because the internal documentation links don't render correctly on GitHub's viewer.

Installation

CMIP Rapid Evaluation Framework can be installed with pip, mamba or conda:

The following commands don't work yet, but will be updated when we have a release.

pip install cmip-ref
mamba install -c conda-forge cmip-ref
conda install -c conda-forge cmip-ref

For developers

For development, we rely on uv for all our dependency management. To get started, you will need to make sure that uv is installed (instructions here).

We use our Makefile to provide an easy way to run common developer commands. You can read the instructions out and run the commands by hand if you wish, but we generally discourage this because it can be error prone.

The following steps are required to set up a development environment. This will install the required dependencies and fetch some test data, as well as set up the configuration for the REF.

make virtual-environment
uv run esgpull self install $PWD/.esgpull
uv run ref config list > $PWD/.ref/ref.toml
export REF_CONFIGURATION=$PWD/.ref
make fetch-test-data
uv run ref datasets ingest --source-type cmip6 $PWD/.esgpull/data

The local ref.toml configuration file will make it easier to play around with settings. By default, the database will be stored in your home directory, this can be modified by changing the db.database_url setting in the ref.toml file.

The test suite can then be run using make test. This will run the test suites for each package and finally the integration test suite.

If there are any issues, the messages from the Makefile should guide you through. If not, please raise an issue in the issue tracker.

For the rest of our developer docs, please see .