LORIS-MRI Release v22.0.1

This release of the LORIS-MRI repository accompanies the LORIS 22.0.1 release.

This release fixes a bug in the python BIDS insertion scripts. The regular expression to determine a file type is less permissive than before and does only check the extension of the file now instead of the whole path. See #512

LORIS-MRI Release v21.0.5

This release of the LORIS-MRI repository accompanies the LORIS 21.0.5 release.

This release fixes a bug introduced for phantom dataset where the candidate validation was done before checking if the scan was a phantom scan. The issue only happened when loading phantom scans via the insertion scripts.

LORIS MRI Release v22.0.0

This release of the LORIS-MRI repository accompanies the LORIS 22.0.0 release.

• List of all LORIS-MRI 22.0.0 changes (not including schema and module changes which were contributed to the main LORIS repo)

New Features

• Can now import MRI datasets already organized according the the BIDS standard (#499)

• Gzipping of the NIfTI files created by mnc2nii when createNifti option is set to Yes (#468)

Improvements

• Updated the Mac install README to more recent version of LORIS-MRI (#505)

• Removed the site question from the install script since this information is not used anywhere during the install process (#503)

• Updates to the documentation of LORIS-MRI (#502)

• Refactoring of how the config settings are fetched, for those of type string or boolean (#500 & #466)

• Removed the mri_acquisition_dates table and updated the queries associated to that table (#463)

• Exit the DICOM archive insertion if the insertion in the tarchive derived tables failed (#464)

• Added chunking of EDF files for future visualization of EEG datasets via the browser (#462)

• Simplification of the candidate validation code (#461)

• Fixed the query logic to select files for which NIfTI files should be created in mass_nii.pl (#458)

• Stops inserting a dump of the whole MINC header in the parameter_file table (#453)

• Bulletproof the way the host's IP address is computed in dicomTar.pl (#452)

• Fixed get_dicom_file.pl failure when the series description contained a sed special character (#434)

Notes for Existing Projects

• New dependencies (#499) were added to be able to create a pic of the NIfTI files imported via the BIDS structure. To install those dependencies, make sure to first source the LORIS-MRI python virtual environment (source /data/%PROJECT%/bin/mri/python_virtualenvs/loris-mri-python/bin/activate) and run the following commands (without sudo):

pip install matplotlib
pip install nose
pip install sklearn
pip install nilearn

• The following tool script can be run if you wish to gzip the NIfTI files already created by mnc2nii: tools/gzip_nifti_files.pl (#468)

• The profileTemplate.pl has been modified to remove the call to the function NeuroDB::MRI::subjectIDIsValid. This function has been removed from the MRI.pm library since that code was duplicated in MRIProcessingUtility.pm (#461). Ensure to remove the following from your profile file (typically called prod under dicom-archive/.loris-mri) if it was still there.

        # Note, this function will ensure that the PSCID/CandID information refers to a valid candidate
        # and it will check that the Visit label exists in the Visit_Windows table
        if(!NeuroDB::MRI::subjectIDIsValid($subjectID{'CandID'}, $subjectID{'PSCID'}, $subjectID{'visitLabel'}, $dbhr)) {
            return undef;
        }

• If you encounter the following error when running the perl imaging pipeline, update your DBI CPAN library to the latest version with cpan DBI (Reference).

Can't locate object method "selectall_array" via package "DBI::db" at /data/ccna/bin/mri/uploadNeuroDB/NeuroDB/MRIProcessingUtility.pm line 1819.

Clean up

• If you wish to delete all header entries (a.k.a. the full dump of the MINC header) from the parameter_file table, run the clean up patch called 2019_07_04_remove_header_row_from_parameter_file_and_convert_back_to_Value_field_to_text.sql

• If you wish to clean up and remove the mri_acquisition_dates table from your database, run the clean up patch called 2019-07-17_remove_mri_acquisition_dates_table.sql

LORIS MRI Release v21.0.0

This release of the LORIS-MRI repository accompanies the LORIS 21.0.0 release.

• List of all LORIS-MRI 21.0.0 changes (not including schema and module changes which were contributed to the main LORIS repo)

New Features

• Replaced Gender field with Sex in the candidate and tarchive tables (see #336, #382)

• New script to Delete uploaded scans (see #391 ), in the tools directory

• New script to generate BIDS-compliant structures for existing NiFTi files stored in your LORIS instance - tools/MakeNiiFilesBIDSCompliant.pl (see #198)

• Added the image_type header into the mri_protocol table for protocol discrimination of acquisitions. This header is particularly useful to differentiate phase and magnitude files of fieldmap acquisitions (the only difference between the two image types is in the image_type header) (see #416)

• Addition of a tool that produces 3D rendering of the face for visual QC of defaced images (see #415)

• Added a batch script to run defacing in parallel on multiple CPUs using QSUB if the Config Setting is_qsub is set (see #414)

• Grep the age of a candidate at the time of acquisition from the BIDS sub-XXX_scans.tsv file and insert it into physiological_parameter_file (see #447)

Improvements

• Cleaned up multiple database paths referring to the same path (see #399)

• Replaced CenterID field of the candidate table by RegistrationCenterID (see #341, #442)

• Refactoring of database calls using Moose (see #282, #295, #303, #311, #330)

• Replaced all hardcoded prod file mentions with (pre-existing) Config setting (see #254)

• Removed the unused column PendingStaging from the files table (see #292 and LORIS patch aces/Loris#3569)

• Updated the exit code of updateMRI_Upload.pl to use the ExitCode.pm (see #305)

• Harmonized the multiple ways of getting MINC header information in the code (see #351)

• Added missing CPAN dependencies in the install script of LORIS-MRI (see #439)

• Reordering of the environment file for new installs (see #148)

• Upgraded the BIDS insertion pipeline to use the more recent version of pybids (0.9.1) (see #448)

Notes for Existing Projects

Make sure to run the release patch on the LORIS side as there are significant SQL changes affecting the MRI side of LORIS.
Many core Loris-MRI scripts have now been updated with improved command-line arguments (see notes below.)

• Caveat for custom tools: replace Gender with Sex when querying the candidate or tarchive table

• Caveat for custom tools: replace CenterID with RegistrationCenterID when querying the candidate table

• Note Config path changes: Cleaned up multiple database paths referring to the same path (see #399)

• Make sure that your Config setting dataDirBasepath is configured to the proper data path (typically /data/%PROJECT%/data)

• Caveat for projects using the BIDS insertion pipeline. Since pybids changed the way files are queried in the most recent versions, the insertion LORIS-MRI BIDS insertion pipeline had to be refactored to work with the latest stable version of pybids (0.9.1). To upgrade your install of pybids, simply run in your python virtualenv (a.k.a. after having sourced your virtualenv: source $LORIS_MRI/python_virtualenvs/loris-mri-python):

pip uninstall pybids
pip install --upgrade pip
pip install pybids==0.9.1

• New CPAN dependencies to install:

sudo cpan install Moose
sudo cpan install MooseX::Privacy
sudo cpan install TryCatch
sudo cpan install Throwable

• Projects that plan on using the defacing scripts should reorder their environment files as shown in PR #429.

• The following scripts must now be run with the -profile <filename> option set (filename typically prod)

  • batch_upload_tarchive.pl
  • tools/seriesuid2fileid.pl

• Projects that call one of the following scripts using a cronjob should update their cronjob to add the .pl extension to the script:

  • batch_uploads_imageuploader => batch_uploads_imageuploader.pl
  • batch_uploads_tarchive => batch_uploads_tarchive.pl
  • tarchiveLoader => tarchiveLoader.pl
  • uploadNeuroDB/bin/Mincinfo_wrapper => uploadNeuroDB/bin/Mincinfo_wrapper.pl
  • uploadNeuroDB/bin/mincpik => uploadNeuroDB/bin/mincpik.pl
  • tools/seriesuid2fileid => tools/seriesuid2fileid.pl

• Before running the deletion script on an old upload (data uploaded before release 20.3), make sure that the script called cleanup_paths_of_violation_tables.pl (in the tools directory) from #403 (release 20.3) has been run to update all paths in mri_violations_log, mri_protocol_violated_scans and MRICandidateErrors to their actual path in the trashbin subdirectory of your LORIS-MRI data directory.

LORIS MRI Release v20.3.0

This release of the LORIS-MRI repository accompanies the LORIS 20.3.0 release.

• List of all LORIS-MRI 20.3 changes (not including schema and module changes which were contributed to the main LORIS repo)

New Features

• New tool run_defacing_script.pl to deface structural MINC images already registered in LORIS, and register the defaced images back into LORIS (#366 with SQL changes in aces/Loris#4232), using a modified version of the deface_minpipe.pl MINC tool that performs the defacing algorithm developed in the Collins lab at the MNI. (see #366)
• Images with no acquisition date (e.g. in the image header) can now be inserted. This is a new feature to facilitate Open Science projects where no dates should be available in the images' header. ( #395)

Improvements

• [Bug fix] Removed the option -clobber from the call to dicomTar.pl in ImagingUpload.pm so that DICOM archive cannot be overwritten by default by the insertion pipeline (see #396)

• [Bug fix] Use UploadID to update the mri_upload table instead of the TarchiveID when running the insertion pipeline (see #397)

• [Bug fix] Fix the paths stored in the violation tables (mri_violations_log, mri_violated_scans, MRICandidateErrors, see #403)

• [Bug fix] Added a check to verify if CandID and PSCID from the image's header are referring to the same candidate (see #407)

• [Bug fix] Circumvented bug that made tar fail on NFS file systems in get_dicom_script.pl and added an option -id (see #409)

• [Bug fix] Fixed the profileTemplate to include a call to the NeuroDB::MRI:: subjectIDIsValid function to validate subject ID information in getSubjectIDs of the profile file. (see #411)

Notes for Existing Projects

• Make sure to run the tool script called cleanup_paths_of_violation_tables.pl in order to update the MINC location in the 3 violation tables (mri_violations_log, mri_violated_scans, MRICandidateErrors, see #403)

• Because of the missing check (#407) between CandID and PSCID in the image header, it is possible that MINC files are not associated to the correct candidate. We are working on an MRI doctor script that will do all sort of data integrity checks and this will definitely be one of them. (see #400 for details)

• Recommended improvement: Validate your visit labels before loading scans: If the function getSubjectIDs in your prod file (i.e. the MRI configuration file in dicom-archive/.loris-mri/) contains the following flag $subjectID{'createVisitLabel'} set to 1 ($subjectID{'createVisitLabel'} = 1;), then we recommend adding a call to the NeuroDB::MRI:: subjectIDIsValid function to validate the visit label, otherwise the pipeline will automatically create the visit in the session table (see #411 for details).

• If you desire to use the new defacing tool, make sure you have:

  • downloaded and installed the bic-mni-models anatomical template library and the beast library from the MINC tools in /opt/minc/ (https://bic-mni.github.io/#data-files-and-models)
  • added the two following variables to your environment file so that the scripts can find the MNI models (replace %MINC_TOOLKIT_DIR% by the path to your MINC toolkit install):

  # for the defacing scripts
  export BEASTLIB=%MINC_TOOLKIT_DIR%/../share/beast-library-1.1
  export MNI_MODELS=%MINC_TOOLKIT_DIR%/../share/icbm152_model_09c
  

  • sourced the SQL patch from the aces/Loris#4232 LORIS PR

  • adapted the constant variables (%SPECIAL_ACQUISITIONS_FILTER, @MULTI_CONTRAST_ACQUISITIONS_BASE_NAMES) at the top of the tools/run_defacing_script.pl script to your project's need.

  • make sure which deface_minipipe.pl is returning the following $LORIS-MRI/uploadNeuroDB/bin/deface_minipipe.pl script and not the one present in the MINC toolkit (/opt/minc/$VERSION) as LORIS-MRI is using the version of deface_minipipe.pl under development (which have not yet been released with the MINC tools). If which deface_minipipe.pl is not pointing to the correct script, you will need to update the $PATH bash variable of your environment file to make sure $LORIS-MRI/uploadNeuroDB/bin is the first path mentioned in the variable.

LORIS MRI Release v20.2.0

This release of the LORIS-MRI repository accompanies the LORIS 20.2.0 release.

• List of all LORIS-MRI 20.2 changes (not including schema and module changes which were contributed to the main LORIS repo)

New Features

• EEG BIDS import pipeline scripts, written in python (#337, #347)

• Allow insertion of MINC files without associated DICOM files, or TarchiveID or UploadID values (#340): as of this release, it is possible to insert directly MINC files into the database without associated DICOM files using minc_insertion.pl directly.

• New generic (non-MINC) file image insertion script (#271): this script allows insertion of any non-MINC file image into the database via the script imaging_non_minc_insertion.pl to insert non-MINC files into the files table.

• New tool to select, extract and tar DICOM files for a specified regex matching the PatientName and/or acquisition protocol (#360)

• Added a check for the SeriesUID/echo_time pair before insertion into the database to avoid duplication of files in the database. If a file is already registered in the database with that same SeriesUID/echo_time pair of values, the insertion script will return a message saying the image is already inserted in the database with the name of the registered file (see #359 for more details)

Improvements

• Non DICOM files are now ignored when running the default imaging insertion (MRI) pipeline (see #371 and #373)

• Fixed a bug where a DICOM without the header field (0020,0032) was automatically excluded by get_dicom_info.pl because if was flagged as non-DICOM. With this update, running this script with the option -slicepos (#365) will include and load DICOMs that do not contain this header field.

• New Configuration module field under the Imaging pipeline section called "Modalities on which SNR should be calculated (compute_snr_modalities)". This replaces the getSNRModalities() function in the prod (profile configuration) file (#358)

• Fixed a bug when running the DTIPrep pipeline on datasets where NIfTI files are created at the time of insertion (before the bug fix, if the T1W file selected was the NIfTI file instead of the MINC file, then the MINC tools would fail running and the DTIPrep pipeline crashed; this has been fixed in #355).

• Fixed an error message that was displayed in the terminal when loading NeuroDB/MRI.pm library (#357)

• Uses perl function getpwuid instead of whoami fetch the Linux User ID (#368)

• Documentation for more perl scripts: dicomSummary.pl (#342) and get_dicom_info.pl (#343)

• Fixed cleanupTarchives.pl tool and documented it (#344)

• Fixed updateHeaders.pl tool and documented it (#345)

Notes for Existing Projects

• Important: Update your Configuration module, Imaging section -- New field: "Modalities on which SNR should be calculated (compute_snr_modalities)" AND remove from your prod file the function called getSNRModalities() (see #358)

• To use the EEG BIDS pipeline, simply follow the install instructions in #337, #347. For detailed information, consult the updated README.md and 02-Install.md under docs/

• To use the new generic (non-MINC) file image insertion script (#271), you will need to install the JSON CPAN library (may require sudo) :

cpan install JSON

LORIS MRI Release v20.1.0

This release of the LORIS-MRI repo accompanies the LORIS 20.1.0 release.

• List of all LORIS-MRI 20.1 changes (not including schema and module changes in the main LORIS repo)

New Features

• Added phantom validation to the imaging insertion pipeline, matching the DICOM header PatientName field against the appropriate phantom regex stored in the Config module (LegoPhantomRegex / LivingPhantomRegex) (see #338)

Improvements

• Creation of NIfTI .bvec and .bval files when converting MINC files to NIfTI for DWI acquisitions (see #335)

• Moved filterParameters() function out of the profileTemplate (prod) file since this function has never been customized by any project to our knowledge (see #309)

• Cleanup:

• Removal of deprecated tools from the LORIS-MRI repository (see #332)
• Removal of the creation of site-specific incoming directories from the install script since those folders are never used (see #334)
• Moved some scripts from dicom-archive to the tools directory (see #333)

Notes for Existing Projects

• Projects that have the option create_nii set to Yes in the Config module can mass-create all the missing NIfTI .bvec and .bval files for DWI acquisitions by running the create_nifti_bval_bvec.pl script present in the tools directory (see "Notes for existing projects" described in PR #335)

• Make sure to update the $PERL5LIB variable to add the path to your dicom-archive directory so that scripts moved to the tools directory can load perl libraries from dicom-archive/DICOM (see #333)

export PERL5LIB=/data/%PROJECT%/bin/mri/uploadNeuroDB:/data/%PROJECT%/bin/mri/dicom-archive:$PERL5LIB

• Existing projects can remove the filterParameter() function from your LORIS-MRI prod config file since it is not used anymore (see #309)

• For projects currently using site-specific incoming data directories, PR #334 should not impact this practice

LORIS MRI Release v20.0.0

This release of the Loris-MRI repo accompanies the LORIS 20.0 release.

• List of all Loris-MRI 20.0 changes (not including schema and module changes in main LORIS repo)

New Features

Documentation

• New: Complete documentation of LORIS-MRI setup, configuration, including scripts, tools, and database tables, and interaction with the LORIS front-end imaging modules. This documentation will be maintained up-to-date with each release to reflect the release codebase and features.

This documentation includes coverage of:

• Troubleshooting Guide covering commonly-encountered errors when running the LORIS-MRI scripts are introduced #269, #273

This release also includes:

• A tool for auto-documentation of all LORIS-MRI perl scripts and libraries (mass_perldoc_md_creation.pl), designed to generate/update perldoc markdown files with each release #314

Other Features

• Exclude specific scan types from database insertion as MINCs via a new setting in the LORIS Configuration module, under section: Imaging Pipeline (Existing projects -- see notes below for more information.) #319, #321

Improvements

• comma-separated values (or ranges) are no longer permitted in the As a first step in the mri_protocol and mri_protocol_checks tables, as the first step in a multi-step process to normalize the protocol-related database tables.The insertion scripts will exit if they do #283, #318, & #327
• As a first step in a multi-step process to normalize the protocol-related database tables, the mri_protocol and mri_protocol_checks tables no longer allow comma separated values. The insertion scripts will exit if they do #283, #318, & #327
• no_nii configuration setting renamed to create_nii to make it more intuitive for users. This setting appears in the LORIS Configuration module, section: Imaging Pipeline (Existing projects -- see notes below for more information.) #294
• Links to all scripts used in the insertion pipeline scripts have been moved to environment file variables #320
• Improved logic in MRI.pm:getPSC()-- CenterID is returned first from the session table if possible, before regex matching in thePatientName` field. #317
• No more jiv files generated by the insertion scripts. (Formerly, these files were used for BrainBrowser visualization, which also supports MINC) #310
• Bug fix in the removal of the trailing / in the $data_dir path variable found in DTIPrep_pipeline, DTIPrepRegister and database_files_update.pl #307
• No more warning about uninitialized value $oldCount when running tarchiveLoader #329
• Removal of the extra unnecessary parameter that was passed to validateCandidate in the MRIProcessingUtility.pm #316

Notes for Existing Projects

• To generate or viewperldoc documentation for scripts - please install the following dependencies:

     sudo -S cpan install Pod::Perldoc
     sudo -S cpan install Pod::Markdown
     sudo -S cpan install Pod::Usage

After installing these dependencies, use the perldoc feature to view documented scripts in a terminal, and/or generate documentation using pod2markdown #315

• jiv file generation is no longer supported in the insertion pipeline. No action is necessary to be taken by existing projects. However, projects that wish to remove the jiv-related entries from their parameter_file table and move their jiv directory into the archive/ path can run the following script remove_jiv_data_from_db_and_filesystem.pl in the MRI tools/ directory. jiv files can be deleted if desired. #310

• Projects should modify their environment file to include /data/%PROJECT%/bin/mri/uploadNeuroDB/bin in the export PATH line. (%PROJECT% should be replaced by your own directory name.) #320

• Projects can now choose to customize the acquisition series they would like excluded from the DICOM-to-MINC conversion and protocol verification steps (limited and hard-coded to localizers up until this release). Case-insensitive regular expression matching is performed on the Series Description fields as customized by the project in the excluded_series_description accessible from the LORIS Configuration Imaging Pipeline section #319, #321

• The mri_protocol and mri_protocol_checks tables no longer allow comma separated values. The insertion scripts will exit if they do (#283, #318). A LORIS script is provided with this release on the LORIS side to migrate comma-separated values (aces/Loris#3523).

• Note that running the 20.0 LORIS release patch will rename your Config setting no_nii to create_nii, and will update your value for this setting (0/1) automatically. (#294)

LORIS MRI Release v19.1.0

This release of the Loris-MRI repo accompanies the LORIS 19.1 release.

• List of all Loris-MRI 19.1 changes (not including schema and module changes in main LORIS repo)

New Features

• Exit codes are now harmonized across the imaging pipeline insertion scripts, and defined in the new class uploadNeuroDB/NeuroDB/ExitCodes.pm #270, #289, #290, #297

Improvements and Documentation

• Removal of unused scripts and files (Makefile.PL, LDZFix.pl, Makefile from both uploadNeuroDB/ and dicom-archive-tools/ directories, dcm_archive_table.sql, find_uploads_tarchive, CiverRegister.pl, registerFile.pl) #259
• FileType in the files table is now obtained from the ImagingFileType table, instead of being hardcoded in the uploadNeuroDB/NeuroDB/File.pm #266
• minc_insertion.pl can now take in -uploadID as an argument. Either -uploadID or -tarchivePath can be set, but not both simultaneously #275
• Improved logic and messaging when running minc_insertion.pl #285
• DICOM study validation using dcmdump now runs on either PatientName or PatientID depending on the configuration option set for lookupCenterNameUsing. In addition, filenames in the uploaded DICOM study are now escaped when invoking the dcmdump command #291

Notes for Existing Projects

• Projects may wish to check the new harmonized Exit Codes in uploadNeuroDB/NeuroDB/ExitCodes.pm. These new Exit Codes will be visible in LORIS in the Server Process Module for all projects relying on Imaging Uploader in Auto-Launch mode.
• Projects may wish to check the removed and unused scripts in #259 (in case Projects customized those independently)

LORIS-MRI Release v19.0.0

This release of the Loris-MRI repo accompanies the LORIS 19.0 release.

• List of all Loris-MRI 19.0 changes (not including schema and module changes in main Loris repo)

Loris-MRI 19.0 absorbs the former DICOM Archive Tools repository for LORIS.

Existing Projects should not update their code without backing up their current dicom-archive/ directory (see Upgrade Process notes below).

New Features

• mri_protocol_violated_scans table now includes TarchiveID linked to the DICOM tar #211, LORIS#3354

Install

• Database Config table settings are populated by the imaging install script, (#272) for the pipeline, imaging data and imaging code settings (#272), if not already configured manually
• Newer MINC toolkit installations covered in updated README (#260)

Improvements and Documentation

• PatientName header no longer stored in the notification_spool table #201
• Main pipeline script (when running imaging_upload_file.pl) checks UploadID and filepath arguments match the same upload #202
• Improved logic and messaging when cleaning extra files/directories from uploaded DICOM tarballs (e.g. __MACOSX/ .DS_store) #203
• Improved MySQL 5.7 compatibility when DICOM date of birth header is empty #205, and removal of the deprecated ~~ operator from the DTI.pm utility #215
• Protocol identification using mri_protocol table values now checks float number parameters to 4 decimal places #206
• Removed duplicate DBI.pm from dicom-archive/DB/ in favour of existing uploadNeuroDB/NeuroDB/DBI.pm #207
• Foreign key constraints added to multiple imaging-related tables #261, LORIS#3327
• Updated wiki for Imaging setup

Notes for Existing Projects

• Projects may wish to check the notification_spool table for any PatientName header values that shouldn't be there #201

Upgrade Process

Ensure your existing dicom-archive customizations are backed up before updating any code, Due to the integration of the dicom-archive-tools repo into Loris-MRI. For details, see PR #200.

• Back up the entire dicom-archive/ directory, including .loris_mri/ subdirectory (mv recommended, so that code checkout step below will not abort)
• Back up your project-specific environment file
• Update your code: Checkout the latest release or download the release .zip. (Note if dicom-archive/ still exists in your git directory, code checkout will abort.)
• Restore your backed-up .loris_mri/ directory and contents under the dicom-archive/ directory (cp -p recommended to maintain ownership and permission settings)
• Restore your backed-up environment file to its original location (cp -p)