Skip to content

AiiDA Dependency Management

Jump to bottom
Carl Simon Adorf edited this page May 19, 2020 · 44 revisions

test-install

About

This page contains guidelines, concrete procedures, and any other information relevant to the management of dependencies for the aiida-core package and plugins.

General information

The dependencies for the aiida-core package must be managed according to AEP 002.

Dependency Manager (DM):

The current dependency manager should be a member of the dependency-manager team.

Specification of AiiDA core dependencies

The dependencies for the aiida-core package are specified in the setup.json file are divided into install_requires and extras_require categories with various sub-categories.

Furthermore, dependencies are also specified in

  • environment.yml: to create a conda environment
  • docs/requirements_for_rtd.txt: for building documentation
  • pyproject.toml: to specify requirements for building the package
  • requirements/requirements-py-3x.txt: well-defined Python environments for all supported Python versions in compliance with dependencies used for CI testing; these files are automatically maintained by the 'update-requirements' workflow

The specification of a dependency within the setup.json is to be considered authoritative.

In addition to above mentioned files, the DM must also be concerned with the modification of the following files:

  • setup.py: Indirectly specifies install dependencies sourced from setup.json
  • util/dependency_management.py: Utility script used for dependency management related tasks.
  • .github/workflows/ci.yml: Specifies the CI workflow that checks consistency for dependency-specifying files (setup.json, requirements/*.txt, etc.) and carries out tests for environments specified in requirements/*.txt files.
  • .github/workflows/test-install.yml: Specifies the a workflow that tests for the ability to install aiida-core with the given dependencies within the current ecosystem. It also defines a workflow for the semi-automated update of the requirements/*.txt files as part of a pull request. Is executed upon modification of dependencies (setup.json) and nightly.

Utility script

The utils/dependency_management.py script within the aiida-core repository provides commands to validate the consistency of aforementioned files with the setup.json file as well as commands to re-generate them after updating a dependency.

How to modify the dependency specification

Prerequisites

Before adding a new dependency for the aiida-core package, make sure that the dependency adheres to the following requirements:

The newly introduced dependency

  • is needed to close a non-trivial feature gap that could not be resolved easily otherwise,
  • supports all Python versions supported by aiida-core (as specified in setup.json),
  • is available on PyPI and conda-forge [1],
  • appears to be within a stable development stage, e.g., has reached a version 1.0 for projects that follow semantic versioning.

[1] In case that a dependency is not yet available through aforementioned channels, but is considered critical enough to be included anyways, the following steps can be taken:

  1. For lightweight dependencies, consider to vendor the package if permissible by license.
  2. Create a request to the current maintainer to make the dependency available open-source/on PyPI/on conda-forge.
  3. Maintain the PyPI project and conda-forge recipes ourselves.

Flow for adding/updating dependencies

This is the standard workflow for updating the aiida-core dependencies:

1. Update the setup.json file:

Modify the affected entries in setup.json and then update all dependent files with

./util/dependency_management.py generate-all

The command will also be executed by the pre-commit hook (if installed).

For packages that are named differently between PyPI and conda-forge, you might need to add an entry to the SETUPTOOLS_CONDA_MAPPINGS variable within the same script.

2. Update the requirements/ files

The requirements/requirements-py-3.?.txt files define the Python pinned environments used for testing and must be consistent with the dependency specification in the setup.json file. There are two ways you can update those files: semi-automatically and manually.

For a semi-automated update (recommended), push the branch that contains the updated dependency directly to the aiida-core package and create a pull request. The GitHub actions bot will then attempt to create a second pull request with the updated requirements/* files into that same branch. Note: This will not work for pull requests from forks.

For a manual update, simply modify the affected entry within all requirements/ files such that they are consistent with the dependency specification, can be installed for all supported Python versions, and do not create any dependency conflicts. For example, if you have a dependency foo~=1.5.0, then the corresponding entry in the requirements file could be foo==1.5.6 or foo==1.7.1, but not foo==1.4.8 or foo==2.0.1.

Continuous Integration

The consistency of the various files with setup.json is checked with a pre-commit hook that is executing the generate-all and the validate-* commands of utils/dependency_management.py. This hook is also executed for all commits as part of the aiida-core CI workflow.

Any changes to files that specify dependencies (setup.py, setup.json) also trigger the test-install workflow, that explicitly checks whether the package can be installed with pip and conda, and whether the requirements/ files are consistent with the dependency specification. In addition, the test-install workflow is executed nightly to ensure that changes within the Python ecosystem do not lead to issues with installation or cause tests to fail. Finally, the test--install workflow will also attempt to semi-automatically update the requirements/*.txt files in case that inconsistencies are detected by creating a separate pull request into the branch that modifies the dependency specification.

Problematic dependencies

This is a list of dependencies that might be problematic for some reason.

Impact on the build and install chain of aiida-core:

  1. Critical disruption/blocker
  2. Major disruption with user impact
  3. Minor disruption without user impact
  4. Concern, but so far no impact.
  • [1] tornado: old major version <5 required, causing issues for example in the context of AiiDA lab
  • [2] pymatgen: Their use of specifying numpy as a requirement in setup_requires in the call to setuptools.setup() is causing issues, because it will trigger an installation (and eventual build!) of numpy within the build environment for pymatgen, which will fail on Python 3.5 with a version of setuptools<42.0.0 and is suspected to cause other potential problems in the future.
  • [2] sqlalchemy-utils: not available through conda-forge
  • [3] Click: Minor release version 7.1 changed the help output formatting and thus broke our tests and CI workflows.
  • [4] shortuiid: A broken release (0.5.1) was removed from PyPI after upload.
Clone this wiki locally