Contributing to the Fermitools
Thank you for contributing to the Fermitools project and making Fermi data analysis even better! This page is meant to assist developers who wish to work on or test the Fermitools package as a whole. If you wish to contribute code to individual repositories you can do so using standard git methodology.
Note: This page is meant to act as a quick development environment setup guide. For best practices and procedures with regards to Fermitools development it is HIGHLY RECOMMENDED that you read the Fermi Developer Documentation.
Developers wishing to develop packages in the Fermitools should follow the workflow outlined below:
Except in cases where developers are debugging issues with the Conda Build packaging process itself, the process illustrated on the right side of the diagram should be followed. In summary,
- Checkout the tools using a
repomaninvocation as outlined above - Make any source code changes in the checked out code
- Compile the code using SCons to make sure that the code compiles without errors
- Check the code into a branch and issue a Pull Request to alert the FSSC that you would like to merge changes back into the
masterbranch of a Fermitools package repository
In order to checkout the multiple repositories which make up the Fermitools you must use the repoman tool set up by Brian Van Klaveran. Repoman is a tool which checks out container packages (such as ScienceTools or GlastRelease) along with their related dependencies. To get repoman, one need only to install the Fermi version of it via pip:
pip install fermi-repoman
It is recommended that you have SSH keys set up for use with your Github account. For instructions on how to do that see the Github documentation on adding a new SSH key to your GitHub account.
If you prefer to use https protocol you will need to specify the remote base you wish repoman to use by either setting the REMOTE_BASE environment variable or providing repoman the flag --remote-base followed by the location of the fermi-lat github organization (i.e. --remote-base https://github.com/fermi-lat).
Repoman has a number of flags which can be passed to it to customize its operation. For the sake of brevity this document will only cover what you need to know to check out the Fermitools, however, if you would like to know more about repoman's operation you can reference the official repoman documentation.
To check out the most recent master branches of all of the Fermitools packages use the following command:
repoman checkout --develop ScienceTools conda
The --develop flag tells repoman to grab the master branch versions of all the ScienceTools constituant packages. In order to grab any changes that are specific to Fermitools (the ScienceTools conda distribution) the conda branch name is appended to the end of the command.
Repoman grabs branches in a superseding order from left to right. The --develop flag tells repoman to start with the master branch of every repository. Adding conda at the end of the command tells repoman to check each repository for a branch named 'conda' (which has been used for conda development/patches by the dev team) and check that out instead if it exists. This is an extremely useful feature in the common case where you may have a test branch (hypothetically named patch-1) located in one or more repositories that you wish to check out if it exists instead of the master branch. In that case you could use the command repoman checkout --develop ScienceTools conda patch-1. The branch hierarchy for that checkout would therefore be patch-1>conda>master.
To compile the tools using SCons after running a repoman checkout one must first install Fermitools dependencies into a conda environment. To do this without installing the tools themselves the following command can be used:
conda create --name dev --only-deps -c conda-forge/label/cf201901 -c fermi fermitools
The above command will create a new environment named dev and install the Fermitools runtime dependencies into it without installing the Fermitools executables or libraries themselves. The installed libraries and headers can be found in the directory
Fermitools is integrated into a distributable package using the Conda Build utility. When developing the Fermitools distribution in a holistic manner it is often very useful to run checkouts and builds entirely via conda build.
To obtain conda build, simply install it via conda install. It is recommended that developers use the version of conda build that is available via the conda-forge channel as opposed to the default conda channel. This version can be obtained via this command:
conda install -c conda-forge conda-build
To build the Fermitools with Conda Build you must first clone the Fermitools-conda git repository located here:
https://github.com/fermi-lat/Fermitools-conda
This repository contains a number of files which tell conda build how to checkout, compile, and package the Fermitools. Each file will be covered briefly here, however, it is highly recommended that any developer of the Fermitools conda distribution familiarize themselves with conda's documentation on building packages.
activate.csh/activate.sh --- These scripts are run when the environment that the Fermitools are installed in is activated. They set a variety of environment variables, alias, and run checks which are required for the safe execution of the Fermitools. There are two versions of this script in order to ensure compatibility with both C/TC Shell and Bash Shell.
meta.yaml --- This file, written in yaml, provides conda build with metadata information on the package it is building along with a list of required dependencies for building and running the produced binaries. More information on the meta.yaml file can be found in the conda build documentation.
build.sh --- This script provides all of the build instructions that conda build executes when run in the recipe directory, including source code checkout, compiler flag setting, and compilation via SCons.
deactivate.csh/deactivate.sh --- These scripts are run when the environment Fermitools is in is deactivated. In a nutshell they undo all of the changes made by the corresponding activate scripts.
After installing conda build and cloning the ScienceTools-conda-recipe repository a build can be executed by entering the repository and running the conda build command:
conda build --python=2.7 -c conda-forge/label/cf201901 -c fermi .
This command tells conda build to pull required dependencies as outlined in the meta.yaml file from the conda-forge/label/cf201901 and fermi channels.
The build takes a little while to execute so sit tight as conda build does its stuff. Conda build will install dependencies (as outlined in meta.yaml), execute the build.sh script, and package the compiled binaries with necessary metadata (again from the meta.yaml file). At the end of the build, should it complete successfully, conda build will display an upload command with a path to the location of the build. Congratulations! You've just built and packaged the Fermitools conda distribution. The upload command will upload the packaged binaries to the Anaconda Cloud. Don't do this unless you know what you are doing and/or have the blessing of the Fermi-LAT software development team.