An important part of the MGI effort is providing infrastructure and tools to enable reproducible research in computational materials science. For reproducible research to become a widely used, repeatable human based process needs to be replaced by automated open-source logging tools. This is especially the case for simulation management, which is often poorly documented and recorded during the development stages of a research project. A good practice is to use a dedicated simulation management tool (SMT) throughout the development process rather than creating an ad-hoc simulation management scheme. Listed below are a number of requirements for an effective SMT.
Sumatra Cloud is a Flask App for displaying and manipulating data generated by the Sumatra simulation management tool. It has a web API and a web front end. The Sumatra Cloud App uses a combination of Flask and MongoDB. Currently the Sumatra client required users to configure and maintain their own databases for recording simulations. This is an enormous hurdle for the casual user. The Sumatra Cloud App aims to make it easy to use Sumatra. The Cloud App will be maintained on a web hosting service like Google Apps or Heroku.
The development repository for the Sumatra Cloud App is on GitHub.
One particular SMT that is currently being evaluated is Sumatra. It is a is a lightweight system for recording the history and provenance data for numerical simulations. It works particularly well for scientists that are in the intermediate stage between developing a code base and using that code base for active research. This is a common scenario and often results in a mode of development that mixes branching for both code development and production simulations. Using Sumatra avoids this unintended use of the versioning system by providing a lightweight design for recording the provenance data independently from the versioning system used for the code development. The lightweight design of Sumatra fits well with existing ad-hoc patterns of simulation management contrasting with more pervasive workflow tools, which can require a wholesale alteration of work patterns. Sumatra uses a straightforward Django-based data model enabling persistent data storage independently from the Sumatra installation. Sumatra provides a command line utility with a rudimentary web interface, but has the potential to become a full web-based simulation management solution.
Ideally, the logging and recording process is entirely automated with the only researcher contribution being a small "commit messages" that logs the researcher's thoughts, reasons and outcomes for running the simulation.
The SMT should be entirely integrated and aware of the common distributed version control (DVC) tools such as Git, Bazaar and Mercurial. The provenance data and simulation data should not be recorded by the version control system, only the SMT project data should be held in version control.
The SMT client should communicate using a REST API completely independent of any backend databases.
The SMT should have a simple local store (dump to JSON) for when the API is unavailable.
Output data files should be hashed to enable effective replication and future regression testing with a continuous integration tool.
Low overhead for integration of low level regression tests with each provenance record.
All dependencies should be automatically recorded as well as uninstalled development repositories that the simulation depends on. This is hard to achieve across multiple language barriers, but one of the most important requirements.
The SMT should be aware of the status of live jobs and send updates via the API.
The SMT needs to be aware of provenance data associated with parallel jobs (such as which nodes are being used) as well as awareness of various queuing systems.
Every record (simulation) should have a unique ID and an associated time stamp.
One particular SMT that is currently being evaluated is Sumatra. It is a is a lightweight system for recording the history and provenance data for numerical simulations. It works particularly well for scientists that are in the intermediate stage between developing a code base and using that code base for active research. This is a common scenario and often results in a mode of development that mixes branching for both code development and production simulations. Using Sumatra avoids this unintended use of the versioning system by providing a lightweight design for recording the provenance data independently from the versioning system used for the code development. The lightweight design of Sumatra fits well with existing ad-hoc patterns of simulation management contrasting with more pervasive workflow tools, which can require a wholesale alteration of work patterns. Sumatra uses a straightforward Django-based data model enabling persistent data storage independently from the Sumatra installation. Sumatra provides a command line utility with a rudimentary web interface, but has the potential to become a full web-based simulation management solution.