[dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch #790

cmadjar · 2022-08-16T22:23:55Z

Description

This inserts into the MRICandidateErrors table if a NIfTI file has a PatientName that does not correspond to a specific candidate and exits as it is done on the perl side. Otherwise, scans would be inserted anyway to the CandID and it could have attached the scans to a wrong candidate...

…atching between DICOMs and NIfTI

* Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit

* Turn off column statistics option when running mysqldump (#771) * turn off column statistics option when running mysqldump * add new option to the script to provide additional mysqldump options * run mass perldoc * fix perldoc * fix perldoc * check if date exists otherwise set it to null when inserting into the files table (#777) * [dcm2bids] Get scanner candidate's RegistrationProjectID based on the scanned candidate's visit ProjectID (#779) * add registrationProjectID to the function creating the candidate scanner * missed function call in base_pipeline * flake8 * Pull 24.0-release to 24.1-release (#788) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * merge 24.0 release to 24.1-release (#791) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit

* Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL

* Turn off column statistics option when running mysqldump (#771) * turn off column statistics option when running mysqldump * add new option to the script to provide additional mysqldump options * run mass perldoc * fix perldoc * fix perldoc * check if date exists otherwise set it to null when inserting into the files table (#777) * [dcm2bids] Get scanner candidate's RegistrationProjectID based on the scanned candidate's visit ProjectID (#779) * add registrationProjectID to the function creating the candidate scanner * missed function call in base_pipeline * flake8 * Pull 24.0-release to 24.1-release (#788) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * merge 24.0 release to 24.1-release (#791) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * Add PhaseEncodingDirection and EchoNumber to files and mri_violations_log insert statements (#785) * modify insertion into the mri_violations_log * modify is_file_unique function perl side * modify minc_insertion.pl script to add PhaseEncodingDirectin and EchoNumber to the file data object - perl side * check uniqueness of file based on seriesUID, EchoTime, EchoNumber and PhaseEncodingDirection in python scripts * populate PhaseEncDir, EchoTime, EchoNumber in insertion to files table for bids_import * set PhaseEncDir and EchoNb to none if not present in JSON file * update file non unique message * insert PhaseEncDir and EchoNb in mri_violations_log and files tables in python pipeline * fix missing parentheses * fix extra brackets in nifti_insertion.py * file matching type with regex instead of text as pybids returns either json or .json etc depending on it version * fix bug * fix phaseencodingdirection and echonumber set to null when inserting into files table for perl pipeline * Add phase encoding direction and echo number to mri protocol and mri protocol violated scans (#782) * add PhaseEncodingDirection and ScanOptions to mri_protocol and mri_protocol_violated_scans queries in the python code * update perl code to add the new fields in mri_protocol and mri_protocol_violated_scans * fix ScanOptions not being inserted correctly * use EchoNumber instead of ScanOptions as more generic * update doc * fix seriesUID not being inserted in mri_protocol_violated_scans * some print to figure out why scan is not inserted? * some print to figure out why scan is not inserted? * fix tiny bug * other prints * other prints * fix comparison of int and str for echo number comparison between protocol and json file content * Merge 24.0-release into 24.1-release (#796) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL

* Turn off column statistics option when running mysqldump (#771) * turn off column statistics option when running mysqldump * add new option to the script to provide additional mysqldump options * run mass perldoc * fix perldoc * fix perldoc * check if date exists otherwise set it to null when inserting into the files table (#777) * [dcm2bids] Get scanner candidate's RegistrationProjectID based on the scanned candidate's visit ProjectID (#779) * add registrationProjectID to the function creating the candidate scanner * missed function call in base_pipeline * flake8 * Pull 24.0-release to 24.1-release (#788) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * merge 24.0 release to 24.1-release (#791) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * Add PhaseEncodingDirection and EchoNumber to files and mri_violations_log insert statements (#785) * modify insertion into the mri_violations_log * modify is_file_unique function perl side * modify minc_insertion.pl script to add PhaseEncodingDirectin and EchoNumber to the file data object - perl side * check uniqueness of file based on seriesUID, EchoTime, EchoNumber and PhaseEncodingDirection in python scripts * populate PhaseEncDir, EchoTime, EchoNumber in insertion to files table for bids_import * set PhaseEncDir and EchoNb to none if not present in JSON file * update file non unique message * insert PhaseEncDir and EchoNb in mri_violations_log and files tables in python pipeline * fix missing parentheses * fix extra brackets in nifti_insertion.py * file matching type with regex instead of text as pybids returns either json or .json etc depending on it version * fix bug * fix phaseencodingdirection and echonumber set to null when inserting into files table for perl pipeline * Add phase encoding direction and echo number to mri protocol and mri protocol violated scans (#782) * add PhaseEncodingDirection and ScanOptions to mri_protocol and mri_protocol_violated_scans queries in the python code * update perl code to add the new fields in mri_protocol and mri_protocol_violated_scans * fix ScanOptions not being inserted correctly * use EchoNumber instead of ScanOptions as more generic * update doc * fix seriesUID not being inserted in mri_protocol_violated_scans * some print to figure out why scan is not inserted? * some print to figure out why scan is not inserted? * fix tiny bug * other prints * other prints * fix comparison of int and str for echo number comparison between protocol and json file content * Merge 24.0-release into 24.1-release (#796) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * set file number to undef if it is '' (#800) * [dcm2bids] Add missing get_scanner_candid function in imaging.py (#801) * add missing function called from database_config.py get_subject_ids function * add python function documentation * Add TaskName field to JSON side car file for modalities with task-* names (#802) * add taskName to JSON side car file for modalities with task-* names * add relevant comments * Add required IntendedFor BIDS field in fieldmap JSON files (#803) * add IntendedFor to fieldmaps automatically add visit_label in intended for paths * add method documentation * fix line too long * fix line too long * Update VERSION to 24.1.0 * Update release version in the README.md file * Fix batch_uploads_tarchive to redirect to dcm2bids pipeline when dcm2niix is set as a converter in the Configs (#804) * modify batch_uploads_tarchive so that it can run dcm2bids or dcm2mnc pipeline depending on the converter mentioned in the Config module * fix typo identified by Christine (#807) * [dcm2bids pipeline] Delete temporary directory created by the pipeline once it is done running (#806) * remove tmp dirs before exiting pipeline * add comments

* Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca>

* Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove print Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com>

* Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (#876) * Merge 24.0.4 into 24.1 release (#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (#876) * Merge 24.0.4 into 24.1 release (#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * add a delete_file function for S3 (#881) * MATLAB 7.3 HDF5 Workaround (#880) * Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic * Fix nii insertion mri protocol group id bug (#883) * fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * Do not register multiple entries of the same violation in `mri_violations_log` and `mri_protocol_violated_scans` (#884) * keep track of additional event data (24.1) (#873) * keep track of additional event data * py eeg/physiological * Copy .set file before updating paths (#886) * change version to 24.1.8 (#887) * correct flake errors * Regis' feedback --------- Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com> Co-authored-by: jeffersoncasimir <15801528+jeffersoncasimir@users.noreply.github.com>

* Turn off column statistics option when running mysqldump (aces#771) * turn off column statistics option when running mysqldump * add new option to the script to provide additional mysqldump options * run mass perldoc * fix perldoc * fix perldoc * check if date exists otherwise set it to null when inserting into the files table (aces#777) * [dcm2bids] Get scanner candidate's RegistrationProjectID based on the scanned candidate's visit ProjectID (aces#779) * add registrationProjectID to the function creating the candidate scanner * missed function call in base_pipeline * flake8 * Pull 24.0-release to 24.1-release (aces#788) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * merge 24.0 release to 24.1-release (aces#791) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit

* Turn off column statistics option when running mysqldump (aces#771) * turn off column statistics option when running mysqldump * add new option to the script to provide additional mysqldump options * run mass perldoc * fix perldoc * fix perldoc * check if date exists otherwise set it to null when inserting into the files table (aces#777) * [dcm2bids] Get scanner candidate's RegistrationProjectID based on the scanned candidate's visit ProjectID (aces#779) * add registrationProjectID to the function creating the candidate scanner * missed function call in base_pipeline * flake8 * Pull 24.0-release to 24.1-release (aces#788) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * merge 24.0 release to 24.1-release (aces#791) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * Add PhaseEncodingDirection and EchoNumber to files and mri_violations_log insert statements (aces#785) * modify insertion into the mri_violations_log * modify is_file_unique function perl side * modify minc_insertion.pl script to add PhaseEncodingDirectin and EchoNumber to the file data object - perl side * check uniqueness of file based on seriesUID, EchoTime, EchoNumber and PhaseEncodingDirection in python scripts * populate PhaseEncDir, EchoTime, EchoNumber in insertion to files table for bids_import * set PhaseEncDir and EchoNb to none if not present in JSON file * update file non unique message * insert PhaseEncDir and EchoNb in mri_violations_log and files tables in python pipeline * fix missing parentheses * fix extra brackets in nifti_insertion.py * file matching type with regex instead of text as pybids returns either json or .json etc depending on it version * fix bug * fix phaseencodingdirection and echonumber set to null when inserting into files table for perl pipeline * Add phase encoding direction and echo number to mri protocol and mri protocol violated scans (aces#782) * add PhaseEncodingDirection and ScanOptions to mri_protocol and mri_protocol_violated_scans queries in the python code * update perl code to add the new fields in mri_protocol and mri_protocol_violated_scans * fix ScanOptions not being inserted correctly * use EchoNumber instead of ScanOptions as more generic * update doc * fix seriesUID not being inserted in mri_protocol_violated_scans * some print to figure out why scan is not inserted? * some print to figure out why scan is not inserted? * fix tiny bug * other prints * other prints * fix comparison of int and str for echo number comparison between protocol and json file content * Merge 24.0-release into 24.1-release (aces#796) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * set file number to undef if it is '' (aces#800) * [dcm2bids] Add missing get_scanner_candid function in imaging.py (aces#801) * add missing function called from database_config.py get_subject_ids function * add python function documentation * Add TaskName field to JSON side car file for modalities with task-* names (aces#802) * add taskName to JSON side car file for modalities with task-* names * add relevant comments * Add required IntendedFor BIDS field in fieldmap JSON files (aces#803) * add IntendedFor to fieldmaps automatically add visit_label in intended for paths * add method documentation * fix line too long * fix line too long * Update VERSION to 24.1.0 * Update release version in the README.md file * Fix batch_uploads_tarchive to redirect to dcm2bids pipeline when dcm2niix is set as a converter in the Configs (aces#804) * modify batch_uploads_tarchive so that it can run dcm2bids or dcm2mnc pipeline depending on the converter mentioned in the Config module * fix typo identified by Christine (aces#807) * [dcm2bids pipeline] Delete temporary directory created by the pipeline once it is done running (aces#806) * remove tmp dirs before exiting pipeline * add comments

* update the VERSION for the next bug fix release (aces#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (aces#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (aces#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (aces#824) * fix some errors when RepetitionTime is not available in JSON file (aces#825) * Add capability to download file from s3 (aces#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (aces#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (aces#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (aces#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (aces#830) * skip violation if not found on filesystem since it means the scan has been rerun (aces#831) * update VERSION file (aces#832) * do not push files to S3 when their path in the DB is already an S3 URL (aces#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (aces#834) * Merge 24.0 release into 24.1 release (aces#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (aces#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (aces#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (aces#846) * fix bug * fix listing of filenames * comment new function * update version file (aces#847) * Chunk creation subprocess failure check (aces#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix aces#843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (aces#849) * remove prints in nifti_insertion_pipeline.py (aces#851) * fix permissoin denied upon deletion of tmp dir (aces#853) * update to next bug fix relesae (aces#854) * fix duplicated protocols error when same scan type returned (aces#856) * Add missing exit codes on the python's side (aces#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (aces#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (aces#860) * fix intended for bug when no acq time available (aces#861) * fix bug for intended for when getting the list of files needed IntendedFor (aces#862) * fix paths when there are not / at the end of the Config (aces#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (aces#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (aces#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (aces#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (aces#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (aces#876) * Merge 24.0.4 into 24.1 release (aces#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (aces#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (aces#872) * update version to 24.0.4 (aces#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * add a delete_file function for S3 (aces#881) * MATLAB 7.3 HDF5 Workaround (aces#880) * Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic * Fix nii insertion mri protocol group id bug (aces#883) * fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * Do not register multiple entries of the same violation in `mri_violations_log` and `mri_protocol_violated_scans` (aces#884) * keep track of additional event data (24.1) (aces#873) * keep track of additional event data * py eeg/physiological * Copy .set file before updating paths (aces#886) * change version to 24.1.8 (aces#887) * correct flake errors * Regis' feedback --------- Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com> Co-authored-by: jeffersoncasimir <15801528+jeffersoncasimir@users.noreply.github.com>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (#876) * Merge 24.0.4 into 24.1 release (#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * add a delete_file function for S3 (#881) * MATLAB 7.3 HDF5 Workaround (#880) * Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic * Fix nii insertion mri protocol group id bug (#883) * fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * Do not register multiple entries of the same violation in `mri_violations_log` and `mri_protocol_violated_scans` (#884) * keep track of additional event data (24.1) (#873) * keep track of additional event data * py eeg/physiological * Copy .set file before updating paths (#886) * change version to 24.1.8 (#887) * fix incomplete format for query to files table for EchoNumber field (#889) * Add a check before inserting in mri_candidate_errors.py (#891) * add a check before inserting in mri_candidate_errors.py * use TE instead of imageType * fix gzip_nifti empty (#898) * Fix bug when scan type is provided to nifti_insertion.py script (#899) * fix NIfTI insertion scan type name attribute not found * fix no scan type id * fix nifti insertion when scan type is not provided (#901) * change Task to Property prefix for additional events (#900) * add image_type to mpvs (#902) * Fix run numbering logic to use SessionID and not TarchiveID to figure out the next number of a run (#903) * fix run number logic to use SessionID instead of TarchiveID to determine run number * fix bug to query the files for the sessionID instead of hte tarchiveid * fix doc * update version to 24.1.9 (#904) --------- Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com> Co-authored-by: jeffersoncasimir <15801528+jeffersoncasimir@users.noreply.github.com> Co-authored-by: Zaliqa <zaliqa.rosli@mcin.ca>

* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (#876) * Merge 24.0.4 into 24.1 release (#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * add a delete_file function for S3 (#881) * MATLAB 7.3 HDF5 Workaround (#880) * Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic * Fix nii insertion mri protocol group id bug (#883) * fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * Do not register multiple entries of the same violation in `mri_violations_log` and `mri_protocol_violated_scans` (#884) * keep track of additional event data (24.1) (#873) * keep track of additional event data * py eeg/physiological * Copy .set file before updating paths (#886) * change version to 24.1.8 (#887) * fix incomplete format for query to files table for EchoNumber field (#889) * Add a check before inserting in mri_candidate_errors.py (#891) * add a check before inserting in mri_candidate_errors.py * use TE instead of imageType * fix gzip_nifti empty (#898) * Fix bug when scan type is provided to nifti_insertion.py script (#899) * fix NIfTI insertion scan type name attribute not found * fix no scan type id * fix nifti insertion when scan type is not provided (#901) * change Task to Property prefix for additional events (#900) * add image_type to mpvs (#902) * Fix run numbering logic to use SessionID and not TarchiveID to figure out the next number of a run (#903) * fix run number logic to use SessionID instead of TarchiveID to determine run number * fix bug to query the files for the sessionID instead of hte tarchiveid * fix doc * update version to 24.1.9 (#904) * get SessioNID from the session_obj instead of the tarchive obj (#909) * Update VERSION to 24.1.10 (#910) --------- Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com> Co-authored-by: jeffersoncasimir <15801528+jeffersoncasimir@users.noreply.github.com> Co-authored-by: Zaliqa <zaliqa.rosli@mcin.ca>

insert into MRICandidateErrors when candidate mismatch or pname not m…

f5b8a6a

…atching between DICOMs and NIfTI

cmadjar added Bug Priority: High Python imaging pipeline labels Aug 16, 2022

cmadjar changed the base branch from main to 24.0-release August 16, 2022 22:24

cmadjar added this to the 24.0.2 milestone Aug 16, 2022

remove exit

84667be

cmadjar merged commit c756b52 into aces:24.0-release Aug 17, 2022

cmadjar added Add to release notes Copied in release notes labels Aug 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch #790

[dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch #790

cmadjar commented Aug 16, 2022

[dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch #790

[dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch #790

Conversation

cmadjar commented Aug 16, 2022

Description