LORIS-MRI Release v18.0.0

This release of the Loris-MRI repo accompanies the LORIS 18.0 release and the current version of the DICOM Archive Tools repository for LORIS

• All changes for Loris-MRI 18.0 (not including schema and module changes in main Loris repo)
• Note this release is tracked to the 17.1-dev GitHub branch

New Features

• Some study configuration settings migrated to the database from the $profile/prod file : #82, aces/Loris#2649, dicom-archive-tools#45. As of Loris 18.0, these settings can be accessed via Configuration module in the front-end (Admin menu) under the 'Imaging Pipeline' section.

• The sample wrapper script deletemincsqlwrapper.pl (released in 17.0: #172) now has the option to re-insert deleted MINC files (#193). This script is also moved to a newly created directory example_scripts within the tools/ directory with a README for more details about the script functionality.

Install

• Improved CentOS install: no more failure due to the DICOM toolkit #188
• Ubuntu MINC toolkit installation command #190
• Improved README #191

Improvements and Documentation:

• Updated wiki for Imaging setup
• Bug fix in the number of MINCs inserted when using minc_deletion.pl : #180, #185, #189
• Better DICOM file validation #178 and more elaborate exclusion list when validating DICOM files #186, dicom-archive-#46
• Better MySQL 5.7 compatibility #192, aces/Loris#2905.

Notes for Existing Projects

• When updating your Loris-MRI code to the release branch, ensure your dicom-archive-tools code is updated at the same time.
• After applying the Loris release patch, migrate settings from your prod file (Section II) to the database Config table. Run the new tools/ProdToConfig.pl script (once only) to perform this migration, check script/terminal output messages and verify the Config table settings afterwards.
  IMPORTANT: See #182 for complete instructions; note additional changes needed to the prod:get_DTI_Site_CandID_Visit routine. Also note that the $bin_dir setting is not migrated from the prod file to avoid duplication (since its value should already match the database ConfigSetting Loris-MRI code).

LORIS MRI Release v17.0.0

This release of the Loris-MRI repo accompanies the LORIS 17.0 release and the current version of the DICOM Archive Tools repository for LORIS

All changes for Loris-MRI 17.0 (not including schema and module changes in main Loris repo)

New Features

• Sample prod file allows new scans to be inserted must match mri_protocol table but are no longer required to match isFileToBeRegisteredGivenProtocol function (dicom-archive-tools#41)
• New command-line tool seriesuid2fileid to compare imaging data on server against database insertion status (for DICOM and minc files), based on SeriesUID #155
• New script minc_deletion.pl to delete a minc file based on SeriesUID #138, and a sample wrapper script deletemincsqlwrapper.pl for calling minc_deletion.pl #172
• New handling of duplicate detection via tarchive_files table now takes into account Echo time as well as SeriesUID (dicom-archive-tools#42, Loris#1978)
• When duplicates are detected, notification spool error message indicates next script to run #165
• Explicit ordering of modalities enabled via new AcqOrderPerModality column in files table Loris#2221
• Scanner ID is more clearly specified via the files table, moved from the parameter_file table to better distinguish human vs. phantom scans (#156, Loris#2189)

Install

• CentOS install process now runs more of the imaging_install.sh script #163

Improvements and Documentation

• Improved integration of Imaging Uploader to insertion pipeline setup
• Improved pipeline status messages and debug messages
• Various bug fixes
• For projects using the imaging uploader's auto-launch insertion feature, log files are deleted only if insertion was successful Loris#2252
• Force-inserting a scan into LORIS (using the acquisition_protocol flag in minc_insertion.pl #141) now automatically sets the Caveat flag to True for this file in Imaging Browser #171

Notes for Existing Projects

When updating your Loris-MRI code to the release branch, it important to ensure your dicom-archive-tools code is updated at the same time.

Note carefully recent updates to dicom-archive-tools/profileTemplate file coinciding with 17.0 release. Note that your prod file should be manually updated with any applicable changes committed to this profileTemplate file. Examples include:

• isFileToBeRegisteredGivenProtocol() change : new scans to be inserted should match mri_protocol table but are no longer required to match isFileToBeRegisteredGivenProtocol function in prod file (dicom-archive-tools#41)

After applying the Loris 17.0 release patch, run the new tools/BackPopulateSNRAndAcquisitionOrder.pl script to back-populate an existing database with SNR (in parameter_file table) and AcqOrderPerModality (in files) table. See #162 for complete instructions, and note that prod:getSNRfunction() is a pre-requisite to back-populating SNR.

Caveats / Known Issues

LORIS Imaging Preprocessing Pipeline 17.0 (Release Candidate 1)

This release of the Loris-MRI repo accompanies the LORIS 17.0 release and the current version of the DICOM Archive Tools repository for LORIS

All changes for Loris-MRI 17.0 (not including schema and module changes in main Loris repo)

New Features

• Sample prod file allows new scans to be inserted must match mri_protocol table but are no longer required to match isFileToBeRegisteredGivenProtocol function (dicom-archive-tools#41)
• New command-line tool seriesuid2fileid to compare imaging data on server against database insertion status (for DICOM and minc files), based on SeriesUID #155
• New script minc_deletion.pl to delete a minc file based on SeriesUID #138
• New handling of duplicate detection via tarchive_files table now takes into account Echo time as well as SeriesUID (dicom-archive-tools#42, Loris#1978
• When duplicates are detected, notification spool error message indicates next script to run #165
• Explicit ordering of modalities enabled via new AcqOrderPerModality column in files table Loris#2221
• Scanner ID is more clearly specified via the files table, moved from the parameter_file table to better distinguish human vs. phantom scans (#156, Loris#2189)

Install

• CentOS install process now runs more of the imaging_install.sh script #163

Improvements and Documentation

• Improved integration of Imaging Uploader to insertion pipeline setup
• Improved pipeline status messages and debug messages
• Various bug fixes
• For projects using the imaging uploader's auto-launch insertion feature, log files are deleted only if insertion was successful Loris#2252

Notes for Existing Projects

When updating your Loris-MRI code to the release branch, it important to ensure your dicom-archive-tools code is updated at the same time.

Note carefully recent updates to dicom-archive-tools/profileTemplate file coinciding with 17.0 release. Note that your prod file should be manually updated with any applicable changes committed to this profileTemplate file. Examples include:

• isFileToBeRegisteredGivenProtocol() change : new scans to be inserted should match mri_protocol table but are no longer required to match isFileToBeRegisteredGivenProtocol function in prod file (dicom-archive-tools#41)

After applying the Loris 17.0 release patch, run the new tools/BackPopulateSNRAndAcquisitionOrder.pl script to back-populate an existing database with SNR (in parameter_file table) and AcqOrderPerModality (in files) table. See #162 for complete instructions, and note that prod:getSNRfunction() is a pre-requisite to back-populating SNR.

Caveats / Known Issues

LORIS Imaging Preprocessing Pipeline 16.1.0

This release of the Loris-MRI repo accompanies the LORIS 16.1.0 release and the current version of the DICOM Archive Tools repository

New Features

• New one-step script for batch loading images through the entire back-end insertion pipeline. #133
• Scans that do not match the defined protocol can now be force-inserted into LORIS. To use this feature, add 2 flags to the minc_insertion.pl call: -bypass_extra_file_checks and -acquisition_protocol $scan_type (e.g. t2w). Examples included in #141
• Added Signal-to-Noise Ratio (SNR) computation to imaging pre-processing pipeline for some modalities (t1, t2, flair, pd). The prod config file can be modified to include/exclude modalities. PRs: Loris#1993, Loris-MRI#142, dicom-archive-tools#37

Install and Documentation

• Detailed configuration added to the Readme to enable use of both front-end Imaging Uploader module and back-end pipeline scripts to load imaging files. #154
• CentOS installations can follow the new CentOS Imaging installation transcript Wiki page
• Cloning master branch (the latest release) is now recommended in the Readme

Improvements

• An alternative to the dcm2mnc converter utility can be specified via the prod config file $converter variable. (e.g. binary called: dcm2mnc_for_philips_scanners) #140
• Improved install script directory and file permissions (also documented in the updated Readme): #147
• Cleanup of spurious files #143, warnings #146, and variables (dicom-archive-tools#39)

Notes for Existing Projects

As usual, all MySQL changes relevant to Loris-MRI and dicom-archive-tools repos are included in the Loris 16.1 release upgrade patch for existing projects.

• New one-step back-end pipeline script for batch insertion of multiple imaging datasets may be of particular interest to existing projects #133
• Added Signal-to-Noise Ratio computation requires a manual change to the prod file. Modalities other than the default (t1, t2, flair, pd) can be added/excluded for SNR computation there.

Caveats / Known Issues

• Some imaging volumes have the same SeriesUID and can be distinguished based on EchoTime, but are currently flagged as duplicates by the imaging pipeline based on SeriesUID alone. For Loris 17.0 (next major release) code changes and table updates will be added to handle this.

LORIS Imaging Preprocessing Pipeline 16.0

This release of the Loris-MRI repo accompanies the LORIS 16.0 release and the current version of the DICOM Archive Tools repository for LORIS

New Features

• Notifications using notification_spool table #104
• Release branches in Loris-MRI repo

Install

New CentOS installations can download and install MINC toolkit and DCMTK (DICOM toolkit) dependencies through these steps

Improvements and Documentation

• Improved integration of Imaging Uploader to insertion pipeline setup
• Data directories now made Apache-writable for Imaging Uploader
• Manual installation of MINC toolkit now recommended; no longer supporting automated installation of outdated minctools package (historically via apt-get install)
• Improved pipeline status messages and debug messages
• Status column names clarified in mri_upload table
• Count of Mincs Inserted bug fixed in mri_upload table #119
• Bug fix to series_description identification (pattern matching against multiple mri_protocol table entries) #127
• Site no longer hard-coded, now taking value from prefix/project variable. dicom-archive-tools#33
• Verbose option set to zero by default for dicomTar.pl. dicom-archive-tools#31
• Bug Fix to make Imaging Uploader Auto-Launch mode work as expected dicom-archive-tools#32

Notes for Existing Projects

Loris-MRI installed code can now be updated to the release branch. It important to ensure your dicom-archive-tools code is updated at the same time. The recommended update procedure now closely matches the main Loris repo update process.

Original data stored under /data/incoming/ will now be deleted instead of retained for backup #111
A copy of the incoming data is kept in the /data/$PROJ/data/tarchive/ folder
Some projects were finding that keeping the originally-transferred dataset under /data/incoming/ created an unnecessary level of data backup, as this function can be served by the tarchive directory (barring any conversion or tarring issues).
To disable deletion of the original data from /data/incoming/, remove imaging_upload.pl:235-247

The Imaging Uploader module requires Apache to write to the Loris imaging data paths. Add user lorisadmin to your Apache group, and ensure group ownership by Apache of the data directories:
Ubuntu: $group = www-data
CentOS: $group = apache

    sudo usermod -a -G $group lorisadmin
    sudo chgrp $group -R /data/$PROJ/data/
    sudo chmod -R g+s /data/$PROJ/data/
    sudo chgrp $group -R /data/incoming/
    sudo chmod -R g+s /data/incoming/

To ensure Apache-writability, it is also recommended to add the following line to your environment file (#134)

    umask 0002

The Imaging Uploader requires use of the MRIUploadIncomingPath Config setting. To update your setting, see suggested MySQL patch (LORIS#1627)

The Imaging insertion pipeline now requires MySQL lorisuser to have Create Temporary Tables privilege (see Loris repo #1743)
Several columns in the mri_upload table have been renamed for clarity:

• IsValidated: now IsCandidateInfoValidated #106 (MySQL change)
• Processed: now InsertionComplete; CurrentlyProcessed: now Inserting #107 (MySQL changes)

Minor bug fix (#127) may warrant updating the mri_upload:number_of_mincInserted column for accuracy.

New Verbose column in notification_spool table #1567 (also note updates to dicom-archive-tools, MySQL)

The Selected column has been moved from the parameter_type table to the files_qcStatus table. Imaging projects should be sure to carefully note, back up and execute the transfer of this valuable QC data between tables.

• Site no longer hard-coded : Update your prod file with site value dicom-archive-tools#33
• Verbose option set to zero by default for dicomTar.pl. dicom-archive-tools#31

MINC utilities

Manual installation of MINC toolkit is now recommended to take advantage of up-to-date utilities.
Prior to release 16.0, the minctools package was automatically installed by the Loris-MRI imaging install script.
Existing projects can continue using the minctools package (e.g. for data continuity reasons).

MINC utilities for BrainBrowser

For all projects using BrainBrowser, MINC utilities must be available to the main LORIS codebase. #109
(If the Loris-MRI codebase is installed on a separate machine, ensure your MINC utilities are installed in both locations.)

Ensure the project/config.xml file (in the main LORIS codebase) contains the following tagset, specifying the MINC utilities path local to the main LORIS codebase (/opt/minc/ in this example):

<!-- MINC TOOLS PATH -->
< MINCToolsPath>/opt/minc/</MINCToolsPath>

Note that existing projects already using minctools on the same machine should set their MINCToolsPath value to /usr/ (or run which mincheader to find the minctools path).

Caveats / Known Issues

New CentOS installations can download and install MINC toolkit and DCMTK (DICOM toolkit) dependencies through these steps

LORIS Imaging Preprocessing Pipeline 15.10

New Features

• NifTI conversion: Imaging pipeline can automatically generate NifTI in addition to MINC format for all incoming DICOM volumes
• PET insertion supported

Notes for existing projects

• Tarchive path is now relative in the tarchive table. Requires updating the dicom-archive-tool repository, and running its MakeArchiveLocationRelative.pl script
• ClassifyAlgorithm field removed from imaging file metadata
• prod config file updates:
  • line 33: set $bin_dir variable to directory where Loris-MRI code is installed
  • Recommended: add NifTI conversion parameter $no_nii = 0; (or =1 to use NifTI)
• environment config file: set $LORIS_CONFIG variable to dicom-archive directory

Caveats / Known Issues

• Install the latest stable minc toolkit available from the BIC website to resolve the following errors thrown by the imaging pipeline:
  • mincpik: Failed executing mincreshape due to Segmentation fault in mincreshape
  • make_jiv error due to missing utilities: Can't locate auto/MNI/MincUtilities/volume_para.al in @ INC ... at /data/$proj/bin/mri/uploadNeuroDB/bin/minc2jiv.pl line 124
    To update your minc toolkit:
  • sudo apt-get remove minc-tools (to remove minc tools installed by imaging_install.sh)
  • Download and follow instructions on the BIC website to install dependencies and the minc toolkit, and update your .bashrc

LORIS Imaging Preprocessing Pipeline 15.10

New Features

• NifTI conversion: Imaging pipeline can automatically generate NifTI in addition to MINC format for all incoming DICOM volumes
• PET insertion supported

Notes for existing projects

• Tarchive path is now relative in the tarchive table. Requires updating the dicom-archive-tools repository, and running its MakeArchiveLocationRelative.pl script
• ClassifyAlgorithm field removed from imaging file metadata
• prod config file updates:
  • line 33: set $bin_dir variable to directory where Loris-MRI code is installed
  • Recommended: add NifTI conversion parameter $no_nii = 0; (or =1 to use NifTI)
• environment config file: set $LORIS_CONFIG variable to dicom-archive directory

LORIS Imaging Preprocessing Pipeline 15.04

New Features

Added 4 new perl files:
uploadNeuroDB/imaging_upload_file.pl
uploadNeuroDB/imaging_upload_file_cronjob.pl
uploadNeuroDB/NeuroDB/ImagingUpload.pm
uploadNeuroDB/NeuroDB/FileDecompress.pm

These files incorporate the ability to:

1. Validate and decompress uploaded anonymized images’ archive,
2. Run dicomTar.pl and tarchiveLoader scripts,
3. Move the uploaded images archive to “/data/incoming” (after completion of the imaging pipeline),
4. And update the mri_upload mysql table.
   Note: If the “ImagingUploaderAutoLaunch” config option is set to '1' in the mysql Config table, then the imaging_upload_file_cronjob.pl script will be automatically executed. Otherwise, the imaging_upload pipeline can be launched from the back-end by either:
   adding imaging_upload_file_cronjob.pl to the crontab so that all scans from the uploaded archive can be inserted into the database,
   OR running imaging_upload_file.pl for each uploaded archive of images using the following command:
   imaging_upload_file.pl /path_to_UploadedFile -profile prod -upload_id -verbose

Notes for existing projects

New perl dependencies (command to run):
Archive::Extract (cpan install Archive::Extract)
Archive::Zip (cpan install Archive::Zip)
In the prod file, add the path to the imaging scripts in the '$bin_dir' config variable
In the mri_upload table, the “Processed” field should be set to '1' for all the previously uploaded archives of images. Otherwise, the imaging pipeline will be rerun on these archives of images by the cronjob using imaging_upload_file_cronjob.pl.

LORIS Imaging Preprocessing Pipeline 14.11

New Features

• Added ability to add any form of processed data
• Imaging pipeline scripts have been optimized and modularized
• Improved logging

Notes for existing projects

• New dependency. Must run “cpan install Path::Class”
• Configuration file (prod) moved to new .loris_mri/ directory