Testing

Before making a Pull Request to the ESCMI/CIME repository on GitHub, test your changes.

• Testing changes
  • How to run the tests
    • pytest
      • Examples
        • Run all tests
        • Running tests by category
        • Running a specific test file
        • Running a specific test case
    • scripts_regression_tests.py
      • Examples
        • Running all tests
        • Running tests by category
        • Running a specific test file
        • Running a specific test case
• Code Quality
  • Install pre-commit
  • Run pre-commit
  • Installing git hook scripts
• Docker development/testing container
  • Build the container
    • Customize the container
  • Targets
  • Running the container
    • Environment variables
  • Persisting data
• Continuous Intgeration Testing
• Changes to Fortran or to the test scripts in SystemTests

Testing changes

CIME splits it's testing into two categories unit and sys.

The unit category covers both doctests and unit tests. While the sys category covers regression tests. The tests are named accordingly e.g. unit tests are found as CIME/tests/test_unit*.

How to run the tests

There are two possible methods to run these tests.

pytest

CIME supports running tests using pytest. By using pytest coverage reports are automatically generated. pytest supports all the same arguments as scripts_regression_tests.py, see --help for details.

To get started install pytest and pytest-cov.

pip install pytest pytest-cov
# or
# conda install -c conda-forge pytest pytest-cov

Examples

Run all tests

pytest

Running tests by category

pytest CIME/tests/test_unit*
# or
pytest CIME/tests/test_sys*

Running a specific test file

pytest CIME/tests/test_unit_case.py

Running a specific test case

pytest CIME/tests/test_unit_case.py::TestCaseSubmit::test_check_case

scripts_regression_tests.py

The scripts_regression_tests.py script is located under CIME/tests.

You can pass either the module name or the file path of a test.

Examples

Running all tests

python CIME/tests/scripts_regression_tests.py

Running tests by category

python CIME/tests/scripts_regression_tests.py CIME/tests/test_unit*
# or
python CIME/tests/scripts_regression_tests.py CIME/tests/test_sys*

Running a specific test file

python CIME/tests/scripts_regression_tests.py CIME.tests.test_unit_case

Running a specific test case

python CIME/tests/scripts_regression_tests.py CIME.tests.test_unit_case.TestCaseSubmit.test_check_case

Code Quality

To ensure code quality we require all code to be linted by pylint and formatted using black. We run a few other tools to check XML formatting, ending files with newlines and trailing white spaces.

To ensure consistency when running these checks we require the use of pre-commit.

Our GitHub actions will lint and check the format of each PR but will not automatically fix any issues. It's up to the developer to resolve linting and formatting issues. We encourage installing pre-commit's Git hooks that will run these checks before code can be committed.

Install pre-commit

pip install pre_commit
# or
# conda install -c conda-forge pre_commit

Run pre-commit

pre-commit run -a

Installing git hook scripts

If you install these scripts then pre-commit will automatically run on git commit.

pre-commit install

Docker development/testing container

GitHub actions runs all CIME's tests in containers. The dockerfile can be found under the docker/ directory.

You can skip building the container and use the same container from the GitHub actions using the following commands. This will pull the latest image, see the available run modifiers to customize the container.

docker run -it jasonb87/cime:latest bash

This container is built for development and testing purposes.

The default base image is version 4.11.0-0 of condaforge/mambaforge.

The supported compiler is gnu provided by conda-forge.

The OpenMPI version of libraries is installed by default.

Build the container

docker build -t {image}:{tag} --target {target} docker/

# e.g. building the base image
docker build -t cime:latest --target base docker/

Customize the container

When building the container some features can be customized. Multiple --build-arg arguments can be passed.

docker build -t {image}:{tag} --build-arg {name}={value} docker/

Argument	Description	Default
MAMBAFORGE_VERSION	Version of the condaforge/mambaforge image used as a base	4.11.0-0
PNETCDF_VERSION	Parallel NetCDF version to build	1.12.1
LIBNETCDF_VERSION	Version of libnetcdf, the default will install the latest	4.8.1
NETCDF_FORTRAN_VERSION	Version of netcdf-fortran, the default will install the latest	4.5.4
ESMF_VERSION	Version of ESMF, the default will install the latest	8.2.0

Targets

There are three possible build targets in the Dockerfile. The slurm and pbs targets are built on top of the base.

When running either slurm or pbs it's important to use the --hostname docker argument since both batch systems are looking for a host named docker.

Target	Description
base	Base image with no batch system.
slurm	Slurm batch system with configuration and single queue.
pbs	PBS batch system with configuration and single queue.

docker build -t {image}:{tag} --target {target} docker/

# e.g building the slurm image
docker build -t cime:latest --target slurm docker/

Running the container

The default environment is similar to the one used by GitHub Actions. It will clone CIME into /src/cime, set CIME_MODEL=cesm and run CESM's checkout_externals. This will create a minimum base environment to run both unit and system tests.

The CIME_MODEL environment vairable will change the environment that is created.

Setting it to E3SM will clone E3SM into /src/E3SM, checkout the submodules and update the CIME repository using CIME_REPO and CIME_BRANCH.

Setting it to CESM will clone CESM into /src/CESM, run checkout_externals and update the CIME repository using CIME_REPO and CIME_BRANCH.

The container can further be modified using the environment variables defined below.

docker run -it --name cime --hostname docker cime:latest bash

# Run with E3SM
docker run -it --name cime --hostname docker -e CIME_MODEL=e3sm cime:latest bash

It's recommended when running the container to pass --hostname docker as it will match the custom machine defined in config_machines.xml. If this is omitted, --machine docker must be passed to CIME commands in order to use the correct machine definition.

Environment variables

Environment variables to modify the container environment.

Name	Description	Default
INIT	Set to false to skip init	true
GIT_SHALLOW	Performs shallow checkouts, to save time	false
UPDATE_CIME	Setting this will cause the CIME repository to be updated using CIME_REPO and CIME_BRANCH	"false"
CIME_MODEL	Setting this will change which environment is loaded
CIME_REPO	CIME repository URL	https://github.com/ESMCI/cime
CIME_BRANCH	CIME branch that will be cloned	master
E3SM_REPO	E3SM repository URL	https://github.com/E3SM-Project/E3SM
E3SM_BRANCH	E3SM branch that will be cloned	master
CESM_REPO	CESM repository URL	https://github.com/ESCOMP/CESM
CESM_BRANCH	CESM branch that will be cloned	master

Examples

# Start just the container no checking out code
docker run -it -e INIT=false cime:latest bash

# Checkout a specific branch from a repository
docker run -it -e CIME_REPO=https://github.com/user/cime -e CIME_BRANCH=updates_xyz cime:latest bash

Persisting data

The config_machines.xml definition as been setup to provided persistance for inputdata, cases, archives and tools. The following paths can be mounted as volumes to provide persistance.

• /storage/inputdata
• /storage/cases
• /storage/archives
• /storage/tools

docker run -it -v {hostpath}:{container_path} cime:latest bash

e.g.
docker run -it -v ${PWD}/data-cache:/storage/inputdata cime:latest bash

It's also possible to persist the source git repositories.

docker run -it -v ${PWD}/src:/src cime:latest bash

Local git respositories can be mounted as well.

docker run -v ${PWD}:/src/cime cime:latest bash

docker run -v ${PWD}:/src/E3SM cime:latest bash

Continuous Intgeration Testing

CIME uses Jenkins to run scripts_regression_tests nightly. Latest results are on http://my.cdash.org/index.php?project=CIME.

Changes to Fortran or to the test scripts in SystemTests

If there are changes to Fortran code in CIME or to the scripts in /utils/python/CIME/SystemTests, you must run the cime_developer tests.

1. Run the X and A system tests with ::

   ./create_test cime_developer

Note: The list of cime_developer tests is defined in cime/scripts/lib/update_acme_tests.py

As noted under "Changes to python", the cime_developer system tests are run automatically if you run scripts_regression_tests.py without the --fast argument - so if you have already run that, you do not have to separately run the above command.

2. If any changes have been made to Fortran code in share/ or driver_cpl/, then run cime unit tests: From the top-level directory in cime, follow the instructions in README.unit_testing (currently only works out-of-the-box on yellowstone).

In addition to CIME standalone testing there is testing associated with each of the CIME models - ACME and CESM. The workflows for each modeling group are a little different and are described below:

• CESM System testing workflow