Merge 24.1.8 into main #888

cmadjar · 2023-02-15T16:46:47Z

This merges the latest bug fix release of 24.1 into the main branch.

…n the `converter` Config setting (aces#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary

* Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca>

…t proper logging is done (aces#824)

…es#825)

* add capability to download file from s3 * fix flake8 error

… upload (aces#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload

…sertion.pl (aces#829)

…to make it executable (aces#830)

… been rerun (aces#831)

aces#833)

…stem before adding them to the list of files to push to S3 (aces#834)

* Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (aces#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca>

…es are already pushed to S3 and not on filesystem anymore (aces#846) * fix bug * fix listing of filenames * comment new function

* Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix aces#843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca>

* add some missing exit codes * add some missing exit codes

…tion was an S3 URL (aces#860)

…edFor (aces#862)

Tested on sandbox with Config `data_dir` = `/data/loris/data`

…eline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (aces#867) Tested on HBCD sandbox on the dataset that caused the issue.

…ns of the pipeline are run (aces#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running

… IntendedFor written in them (aces#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print

* Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (aces#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (aces#872) * update version to 24.0.4 (aces#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca>

* Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic

* fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable

…ions_log` and `mri_protocol_violated_scans` (aces#884)

* keep track of additional event data * py eeg/physiological

regisoc · 2023-02-16T03:06:28Z

Tested for BIDS import: do not know if it is worth mentioning but got some repeated warnings during execution.

/opt/hbcd/bin/mri/python/react-series-data-viewer/eeglab_to_chunks.py:9: RuntimeWarning: Data will be preloaded. preload=False or a string preload is not supported when the data is stored in the .set file
  return mne.io.read_raw_eeglab(path, preload=False)
/opt/hbcd/bin/mri/python/react-series-data-viewer/eeglab_to_chunks.py:9: RuntimeWarning: The data contains 'boundary' events, indicating data discontinuities. Be cautious of filtering and epoching around these events.
  return mne.io.read_raw_eeglab(path, preload=False)

regisoc · 2023-02-16T03:13:35Z

Only execution error found here
Change physiological_file_id by str(physiological_file_id).

cmadjar · 2023-02-16T14:27:59Z

Tested for BIDS import: do not know if it is worth mentioning but got some repeated warnings during execution.

/opt/hbcd/bin/mri/python/react-series-data-viewer/eeglab_to_chunks.py:9: RuntimeWarning: Data will be preloaded. preload=False or a string preload is not supported when the data is stored in the .set file
  return mne.io.read_raw_eeglab(path, preload=False)
/opt/hbcd/bin/mri/python/react-series-data-viewer/eeglab_to_chunks.py:9: RuntimeWarning: The data contains 'boundary' events, indicating data discontinuities. Be cautious of filtering and epoching around these events.
  return mne.io.read_raw_eeglab(path, preload=False)

@laemtl are those warnings expected on main? I don't remember...

laemtl · 2023-02-17T15:49:37Z

@laemtl are those warnings expected on main? I don't remember...

@cmadjar @regisoc We do prefer to not preload when possible, since for large files it creates memory issues. This warning is affecting eeglab files where the data is stored in the set file vs stored in a separate fdt file. It is totally fine.

The boundary notice is totally fine too.

* update the VERSION for the next bug fix release (aces#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (aces#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (aces#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (aces#824) * fix some errors when RepetitionTime is not available in JSON file (aces#825) * Add capability to download file from s3 (aces#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (aces#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (aces#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (aces#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (aces#830) * skip violation if not found on filesystem since it means the scan has been rerun (aces#831) * update VERSION file (aces#832) * do not push files to S3 when their path in the DB is already an S3 URL (aces#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (aces#834) * Merge 24.0 release into 24.1 release (aces#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (aces#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (aces#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (aces#846) * fix bug * fix listing of filenames * comment new function * update version file (aces#847) * Chunk creation subprocess failure check (aces#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix aces#843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (aces#849) * remove prints in nifti_insertion_pipeline.py (aces#851) * fix permissoin denied upon deletion of tmp dir (aces#853) * update to next bug fix relesae (aces#854) * fix duplicated protocols error when same scan type returned (aces#856) * Add missing exit codes on the python's side (aces#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (aces#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (aces#860) * fix intended for bug when no acq time available (aces#861) * fix bug for intended for when getting the list of files needed IntendedFor (aces#862) * fix paths when there are not / at the end of the Config (aces#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (aces#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (aces#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (aces#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (aces#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (aces#876) * Merge 24.0.4 into 24.1 release (aces#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (aces#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (aces#872) * update version to 24.0.4 (aces#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * add a delete_file function for S3 (aces#881) * MATLAB 7.3 HDF5 Workaround (aces#880) * Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic * Fix nii insertion mri protocol group id bug (aces#883) * fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * Do not register multiple entries of the same violation in `mri_violations_log` and `mri_protocol_violated_scans` (aces#884) * keep track of additional event data (24.1) (aces#873) * keep track of additional event data * py eeg/physiological * Copy .set file before updating paths (aces#886) * change version to 24.1.8 (aces#887) * correct flake errors * Regis' feedback --------- Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com> Co-authored-by: jeffersoncasimir <15801528+jeffersoncasimir@users.noreply.github.com>

cmadjar and others added 30 commits November 2, 2022 14:18

update the VERSION for the next bug fix release (aces#816)

f471c59

[dcm2bids] Remove hardcoded dcm2niix binary to use the value stored i…

90e7a9a

…n the `converter` Config setting (aces#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary

fix nonetype errors when the visit of a session does not exist so tha…

0614e8c

…t proper logging is done (aces#824)

fix some errors when RepetitionTime is not available in JSON file (ac…

f47421d

…es#825)

Add capability to download file from s3 (aces#826)

f0810cb

* add capability to download file from s3 * fix flake8 error

Upload to S3: support object name starting with s3://bucket_name/ for…

3644f95

… upload (aces#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload

fix database class pselect documentation for the return type (aces#828)

55c0d8c

map scan type to scan type ID when scan type provided to run_nifti_in…

34e8041

…sertion.pl (aces#829)

modify permission of script run_push_imaging_files_to_s3_pipeline.py …

47c797e

…to make it executable (aces#830)

skip violation if not found on filesystem since it means the scan has…

02507c5

… been rerun (aces#831)

update VERSION file (aces#832)

9c376bb

do not push files to S3 when their path in the DB is already an S3 URL (

e62ea82

aces#833)

fix violation files path when checking if the files are on the filesy…

0d27d49

…stem before adding them to the list of files to push to S3 (aces#834)

fix check if file already inserted in DB (aces#845)

a826be3

Fix logic of determining file run number when previously inserted fil…

c9afaa2

…es are already pushed to S3 and not on filesystem anymore (aces#846) * fix bug * fix listing of filenames * comment new function

update version file (aces#847)

17f6652

Revert chunk_pb2.py changes (aces#849)

afe6903

remove prints in nifti_insertion_pipeline.py (aces#851)

cef8f87

fix permissoin denied upon deletion of tmp dir (aces#853)

3af8cf6

update to next bug fix relesae (aces#854)

115283e

fix duplicated protocols error when same scan type returned (aces#856)

e8103e7

Add missing exit codes on the python's side (aces#857)

049414e

* add some missing exit codes * add some missing exit codes

add ignore case to regex (aces#859)

27f7e63

add download from S3 and reupload if file provided to run_nifti-inser…

bd74369

…tion was an S3 URL (aces#860)

fix intended for bug when no acq time available (aces#861)

a1dc483

fix bug for intended for when getting the list of files needed Intend…

bc00fd6

…edFor (aces#862)

fix paths when there are not / at the end of the Config (aces#866)

82c47b8

Tested on sandbox with Config `data_dir` = `/data/loris/data`

cmadjar and others added 14 commits January 12, 2023 17:31

fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pip…

e77d767

…eline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (aces#867) Tested on HBCD sandbox on the dataset that caused the issue.

Properly update mri_upload 'Inserting' column when different sectio…

9f986c8

…ns of the pipeline are run (aces#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running

update version file to 24.1.6 (aces#870)

d53b0a7

Add download from S3 for fmap already pushed to S3 that needs to have…

0bda1a1

… IntendedFor written in them (aces#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print

update version to 24.1.7 (aces#876)

4fd8b48

add a delete_file function for S3 (aces#881)

eb3c8c4

MATLAB 7.3 HDF5 Workaround (aces#880)

5d50f7f

* Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic

Do not register multiple entries of the same violation in `mri_violat…

3fa41f0

…ions_log` and `mri_protocol_violated_scans` (aces#884)

keep track of additional event data (24.1) (aces#873)

3b0b80e

* keep track of additional event data * py eeg/physiological

Copy .set file before updating paths (aces#886)

d608d18

change version to 24.1.8 (aces#887)

0d82861

Merge tag 'v24.1.8' into merge_24.1.8_into_main

a568f9a

cmadjar added this to the N/A milestone Feb 15, 2023

correct flake errors

0c1d6ab

Regis' feedback

2851cc5

cmadjar merged commit b9d7ee4 into aces:main Feb 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge 24.1.8 into main #888

Merge 24.1.8 into main #888

cmadjar commented Feb 15, 2023 •

edited

Loading

regisoc commented Feb 16, 2023

regisoc commented Feb 16, 2023

cmadjar commented Feb 16, 2023 •

edited

Loading

laemtl commented Feb 17, 2023 •

edited

Loading

Merge 24.1.8 into main #888

Merge 24.1.8 into main #888

Conversation

cmadjar commented Feb 15, 2023 • edited Loading

regisoc commented Feb 16, 2023

regisoc commented Feb 16, 2023

cmadjar commented Feb 16, 2023 • edited Loading

laemtl commented Feb 17, 2023 • edited Loading

cmadjar commented Feb 15, 2023 •

edited

Loading

cmadjar commented Feb 16, 2023 •

edited

Loading

laemtl commented Feb 17, 2023 •

edited

Loading