FabSim3 Design Document

FabSim3 design document

Definition

FabSim3 is a software environment for automating complex computational research activities. In particular, it is focused on automating simulation-based workflows that rely on remote computational resources.

Added value of FabSim3

• Enable remote execution of jobs on supercomputers.
• Establish coupling between models using the workflow automation functionalities.
• Organize input, output and environment information, creating a consistent log and making it possible to repeat/reproduce runs.
• Providing support for ensemble calculations using a simple mechanism.

Dependencies

• Python 3.5 or later
• PyYaml
• Fabric version 1 (Python 3 version)

Target architecture

• Installation: any Linux or Mac-based desktop or laptop.
• Execution host: any Linux or Mac-based machine, including clusters and supercomputers. Aim to support cloud infrastructures in the near future.

Structure of the toolkit

FabSim3 is structured around four building blocks, the base toolkit ('base/' directory), the backends providing additional infrastructure capabilities ('backends/' directory), the primitives providing VVUQ functionalities ('primitives/' directory) and the plugins that allow custom-made and complex functionalities that rely on the first three building blocks ('plugins/' directory).

Base

• Core FabSim3 functionality.
• May depend on primitives or backends, but should be able to be installable/runnable without.
• Customization: Not intended to be customized.

Plugins

• Link to specific use cases.
• May not depend on other plugins, may depend on backends, base and primitives.
• Customization: Free to introduce new plugins, and mix and match existing ones (but plugins cannot be interdependent).

Backends

• Infrastructural
• May depend on other backends or base, but not on primitives, plugins.
• Hierarchical dependencies allowed, but should be avoided.
• Customization: Free to introduce new backends, and mix and match existing ones.

Primitives

• Conceptual
• Hierarchical dependencies allowed.
• May depend on other primitives, but not on backends, plugins or base.
• Customization: Free to introduce new primitives, and mix and match existing ones. Strong requirement to avoid external dependencies and retain a strongly application-independent implementation for each primitive.

Hierarchy of FabSim variables and defaults

FabSim environment variables

env.* variables in FabSim should prioritized as follows (bottom priority bottom, top priority top):

1. Variables explicitly specified directly on the command-line (e.g. cores in fab localhost dummy:dummy_test,cores=5).
2. Defaults set in the FabSim command directly called on the command-line (e.g. a plugin-specific command).
3. Defaults set in any FabSim command that is called underneath.
4. Defaults that are loaded in by deploy/machines.py
   1. Defaults loaded in through machines_user.yml
   2. Defaults loaded in through machines.yml