Skip to content
Timothy B. Brown edited this page Nov 9, 2018 · 55 revisions

Welcome to the Washington University CCF HCP Pipelines Singularity wiki!

The HCP Pipelines Singularity repository is for files to help run the HCP Pipelines using Singularity.

This is the place for documentation for using the HCP Pipelines Singularity repository.

Singularity is a container platform specifically designed for scientific and high performance computing (HPC) use cases.

Table of Contents

1. Installing Singularity

Back to Table of Contents

See Singularity Quick Start Installation for further information.

1a. Debian, Ubuntu

The following steps have been testing on Ubuntu 16.04 LTS

$ sudo apt-get install build-essential libarchive-dev

$ wget https://github.com/singularityware/singularity/releases/download/2.5.2/singularity-2.5.2.tar.gz
$ tar xvf singularity-2.5.2.tar.gz

$ cd singularity-2.5.2
$ ./autogen.sh
$ ./configure --prefix=/usr/local
$ make
$ sudo make install

$ which singularity
/usr/local/bin/singularity

$ singularity --version
2.5.2-dist

1b. Fedora, Red Hat, CentOS

$ sudo yum groupinstall "Development Tools"
$ sudo yum install libarchive-devel

$ wget https://github.com/singularityware/singularity/releases/download/2.5.2/singularity-2.5.2.tar.gz
$ tar xvf singularity-2.5.2.tar.gz

$ cd singularity-2.5.2
$ ./autogen.sh
$ ./configure --prefix=/usr/local
$ make
$ sudo make install

$ which singularity
/usr/local/bin/singularity

$ singularity --version
2.5.2-dist

2. Getting the HCP Pipelines Singularity repository

Back to Table of Contents

2a. Using Git

If you have git installed on your system, you can simply clone the HCP Pipelines Singulary repository as follows:

git clone https://github.com/Washington-University/HCPpipelinesSingularity.git

2b. Using wget

If you don't have or don't care to use git, then you can use the following technique to get a copy of the current latest code in the master branch of the repository. This will not allow you to interact with the repository (e.g. check out branches, make and commit modifications, etc.) via git.

$ wget https://github.com/Washington-University/HCPpipelinesSingularity/archive/master.tar.gz
$ tar xvf master.tar.gz
HCPpipelinesSingularity-master/
HCPpipelinesSingularity-master/.gitignore
HCPpipelinesSingularity-master/HCPprocessPipelines
HCPpipelinesSingularity-master/LICENSE.md
HCPpipelinesSingularity-master/README.md
HCPpipelinesSingularity-master/product.txt
HCPpipelinesSingularity-master/show_version
HCPpipelinesSingularity-master/version.txt

$ mv HCPpipelinesSingularity-master HCPpipelinesSingularity

3. Modifying the Singularity recipe file

Back to Table of Contents

There are some files that are required in order to run the HCP Pipelines that cannot be distributed as part of a Singularity container or automatically retrieved from a known distribution location when the Singularity container is created from a recipe file. You will have to get or generate those files yourself, place them in a known location on your system before building a Singularity container image from a recipe, and modify the Singularity recipe file to reflect the location of those files.

Gradient Distortion Coefficients file

The first of these files that cannot be distributed is the file that contains the gradient distortion coefficients for your scanner. The information in that file is considered to be proprietary information owned by the scanner manufacturer. Therefore, we cannot provide that information as part of an already created Singularity container.

Also, because this is proprietary information, gradient coefficients files cannot be made available for general download from a known site. Therefore, we cannot simply put a reference to a known download site in the Sigularity recipe file.

You will need to obtain the gradient distortion coefficients file either from your scanner manufacturer or from the scanner control computer itself. For example, on Siemens scanners, the gradient coefficients file is usually available at C:\MedCom\MriSiteData\GradientCoil\coeff.grad.

As it is currently written, the Singularity recipe file in the HCP Pipelines Singularity repository, HCPprocessPipelines, assumes that you will create a directory somewhere on your system and place any number of gradient coefficients files in that directory. This is so you can build one Singularity container from the recipe file that supports running HCP Pipelines on data collected from a number of different scanners.

Use the following steps to incorporate your gradient coefficients file(s) into the Singularity container that you build.

  1. Create a directory on the system on which you will build the Singularity container (e.g. /home/tbb/gradient_coefficients_files).

    Note that the system on which you will build the Singularity container does not need to be the same one on which you actually run the processing.

  2. Copy your gradient distortion coefficients files into that directory.

    e.g. cp *.grad /home/tbb/gradient_coefficients_files

  3. Edit the HCPprocessPipelines Singularity recipe file in your local copy of the HCP Pipelines Singularity repository.

    • Find the %files section in the Singularity recipe file. It will look something like:

%files      ##########  Need gradient_coefficient_files!!!  &&  freesurfer license key file from https://surfer.nmr.mgh.harvard.edu/fswiki/License
    /home/j/Downloads/xnat_pbs_jobs_container/gradient_coefficient_files/* /export/HCP/gradient_coefficient_files/
    /home/j/Downloads/xnat_pbs_jobs_container/license.txt /export/freesurfer_license/

  • Change the /home/j/Downloads/xnat_pbs_jobs_container/gradient_coefficient_files/* to point to where you have placed your gradient coefficients files on your system.

    That is, substitute the location in which your gradient coefficients files reside for /home/j/Downloads/xnat_pbs_jobs_container/gradient_coefficient_files/*

    For example: /home/tbb/gradient_coefficients_files/*

For example:

/home/tbb/gradient_coefficients_files/* /export/HCP/gradient_coefficient_files/

FreeSurfer License file

Another file which we can not build in to a Singularity container for distribution nor specify a known download location into a Singularity recipe file is a FreeSurfer license file. This is a simple text file that you must obtain by visiting https://surfer.nmr.mgh.harvard.edu/fswiki/License.

On that page, you will find a link labeled "Follow this link to obtain a license key". Follow the link as instructed, and fill out the form with the requested information. When asked for your Operating System/Platform:, be sure to select Linux/Intel. The container will be effectively running CentOS/Linux, so you will be using the Linux version of FreeSurfer.

Once you have filled out the form, select the I AGREE button to agree to the FreeSurfer license. You will be sent a FreeSurfer license file via email. The FreeSurfer license file is a simple text file attached to the email sent to you. It will likely be named license.txt.

Save the license.txt file to a directory on your system (e.g. /home/tbb/FreeSurferLicense/license.txt).

In the %files section of the Singularity recipe file, change the line that reads /home/j/Downloads/xnat_pbs_jobs_container/license.txt /export/freesurfer_license/.

Substitute your license file location for /home/j/Downloads/xnat_pbs_jobs_container/license.txt

For example:

/home/tbb/FreeSurferLicense/license.txt /export/freesurfer_license/

4. Building a Singularity image file to run HCP Pipelines

Back to Table of Contents

Save the above modifications to the HCPprocessPipelines Singularity recipe file, and then issue the following command while in the directory containing the HCPprocessPipelines Singularity recipe file to create the container image. Note, that you must either be the root user or be able to use the sudo command in order to build a Singularity container image from a Singularity recipe file.

$ sudo singularity build HCPprocessPipelines.img HCPprocessPipelines

There should be quite a lot of text feedback from the singularity command as it builds the Singularity container image based on the recipe file. This process is analogous to installing an operating system (CentOS 7) and all the required software for running the HCP Pipelines on a new computer. This process can take an hour or more.

Fairly early in the textual feedback, you may see some warning messages similar to the following:

WARNING: Warning reading tar header: Ignoring malformed pax extended attribute
WARNING: Warning reading tar header: Ignoring malformed pax extended attribute
WARNING: Warning reading tar header: Ignoring malformed pax extended attribute

Note that these warnings do not affect the output. See https://github.com/singularityware/singularity/issues/1669 for further information.

5. Working with your Singularity image file

Back to Table of Contents

5a. Listing the available apps within the container

$ singularity apps HCPprocessPipelines.img
FunctionalPreprocessing
FunctionalPreprocessingCheck
StructuralPreprocessing
StructuralPreprocessingCheck

5b. Getting help for one of the apps

$ singularity help --app <app-name> HCPprocessPipelines.img

e.g.

$ singularity help --app StructuralPreprocessing HCPprocessPipelines.img

5c. Examining the contents of your container

$ singularity shell HCPprocessPipelines.img
...
Singularity HCPprocessPipelines.img:~/projects/HCPpipelinesSingularity> cd /export/freesurfer-6.0
Singularity HCPprocessPipelines.img:/export/freesurfer-6.0> ls
...
	This is where FreeSurfer 6.0 is installed on the container "machine".
...
Singularity HCPprocessPipelines.img:/export/freesurfer-6.0> cat license.txt
...
	This should show the contents of your FreeSurfer 6.0 license file.
...
Singularity HCPprocessPipelines.img:/export/freesurfer-6.0> cd /export/HCP/Pipelines
Singularity HCPprocessPipelines.img:/export/HCP/Pipelines> ls
...
	This is the version of the HCP Pipelines that you are using in your container
...
Singularity HCPprocessPipelines.img:/export/HCP/Pipelines> cd global/config
Singularity HCPprocessPipelines.img:/export/HCP/Pipelines/global/config> ls
...
	This should show your gradient coefficient files along with other HCP Pipelines global configuration files
...
Singularity HCPprocessPipelines.img:/export/HCP/Pipelines/global/config> cd /pipeline_tools/xnat_pbs_jobs
Singularity HCPprocessPipelines.img:/pipeline_tools/xnat_pbs_jobs> ls

Back to Table of Contents