Upload to S3: support object name starting with s3://bucket_name/ for upload #827
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cmadjar
added a commit
that referenced
this pull request
Nov 14, 2022
* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca>
cmadjar
added a commit
that referenced
this pull request
Dec 6, 2022
* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove print Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com>
cmadjar
added a commit
that referenced
this pull request
Dec 21, 2022
* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com>
cmadjar
added a commit
that referenced
this pull request
Jan 18, 2023
* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com>
cmadjar
added a commit
that referenced
this pull request
Jan 24, 2023
* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (#876) * Merge 24.0.4 into 24.1 release (#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com>
cmadjar
added a commit
that referenced
this pull request
Feb 17, 2023
* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (#876) * Merge 24.0.4 into 24.1 release (#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * add a delete_file function for S3 (#881) * MATLAB 7.3 HDF5 Workaround (#880) * Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic * Fix nii insertion mri protocol group id bug (#883) * fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * Do not register multiple entries of the same violation in `mri_violations_log` and `mri_protocol_violated_scans` (#884) * keep track of additional event data (24.1) (#873) * keep track of additional event data * py eeg/physiological * Copy .set file before updating paths (#886) * change version to 24.1.8 (#887) * correct flake errors * Regis' feedback --------- Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com> Co-authored-by: jeffersoncasimir <15801528+jeffersoncasimir@users.noreply.github.com>
zaliqarosli
pushed a commit
to zaliqarosli/Loris-MRI
that referenced
this pull request
Mar 9, 2023
* update the VERSION for the next bug fix release (aces#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (aces#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (aces#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (aces#824) * fix some errors when RepetitionTime is not available in JSON file (aces#825) * Add capability to download file from s3 (aces#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (aces#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (aces#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (aces#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (aces#830) * skip violation if not found on filesystem since it means the scan has been rerun (aces#831) * update VERSION file (aces#832) * do not push files to S3 when their path in the DB is already an S3 URL (aces#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (aces#834) * Merge 24.0 release into 24.1 release (aces#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (aces#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (aces#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (aces#846) * fix bug * fix listing of filenames * comment new function * update version file (aces#847) * Chunk creation subprocess failure check (aces#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix aces#843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (aces#849) * remove prints in nifti_insertion_pipeline.py (aces#851) * fix permissoin denied upon deletion of tmp dir (aces#853) * update to next bug fix relesae (aces#854) * fix duplicated protocols error when same scan type returned (aces#856) * Add missing exit codes on the python's side (aces#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (aces#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (aces#860) * fix intended for bug when no acq time available (aces#861) * fix bug for intended for when getting the list of files needed IntendedFor (aces#862) * fix paths when there are not / at the end of the Config (aces#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (aces#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (aces#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (aces#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (aces#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (aces#876) * Merge 24.0.4 into 24.1 release (aces#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (aces#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (aces#774) * fix regex search for excluded series description patterns (aces#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (aces#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (aces#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (aces#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (aces#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (aces#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (aces#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (aces#872) * update version to 24.0.4 (aces#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * add a delete_file function for S3 (aces#881) * MATLAB 7.3 HDF5 Workaround (aces#880) * Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic * Fix nii insertion mri protocol group id bug (aces#883) * fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * Do not register multiple entries of the same violation in `mri_violations_log` and `mri_protocol_violated_scans` (aces#884) * keep track of additional event data (24.1) (aces#873) * keep track of additional event data * py eeg/physiological * Copy .set file before updating paths (aces#886) * change version to 24.1.8 (aces#887) * correct flake errors * Regis' feedback --------- Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com> Co-authored-by: jeffersoncasimir <15801528+jeffersoncasimir@users.noreply.github.com>
cmadjar
added a commit
that referenced
this pull request
Mar 10, 2023
* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (#876) * Merge 24.0.4 into 24.1 release (#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * add a delete_file function for S3 (#881) * MATLAB 7.3 HDF5 Workaround (#880) * Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic * Fix nii insertion mri protocol group id bug (#883) * fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * Do not register multiple entries of the same violation in `mri_violations_log` and `mri_protocol_violated_scans` (#884) * keep track of additional event data (24.1) (#873) * keep track of additional event data * py eeg/physiological * Copy .set file before updating paths (#886) * change version to 24.1.8 (#887) * fix incomplete format for query to files table for EchoNumber field (#889) * Add a check before inserting in mri_candidate_errors.py (#891) * add a check before inserting in mri_candidate_errors.py * use TE instead of imageType * fix gzip_nifti empty (#898) * Fix bug when scan type is provided to nifti_insertion.py script (#899) * fix NIfTI insertion scan type name attribute not found * fix no scan type id * fix nifti insertion when scan type is not provided (#901) * change Task to Property prefix for additional events (#900) * add image_type to mpvs (#902) * Fix run numbering logic to use SessionID and not TarchiveID to figure out the next number of a run (#903) * fix run number logic to use SessionID instead of TarchiveID to determine run number * fix bug to query the files for the sessionID instead of hte tarchiveid * fix doc * update version to 24.1.9 (#904) --------- Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com> Co-authored-by: jeffersoncasimir <15801528+jeffersoncasimir@users.noreply.github.com> Co-authored-by: Zaliqa <zaliqa.rosli@mcin.ca>
cmadjar
added a commit
that referenced
this pull request
Mar 28, 2023
* update the VERSION for the next bug fix release (#816) * [dcm2bids] Remove hardcoded dcm2niix binary to use the value stored in the `converter` Config setting (#815) * modifies the dcm2niix command to use the Config converter value instead of hardcoding dcm2niix * add a check to make sure the converter is a dcm2niix binary * Pull 24.0.3 in 24.1 release (#820) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * fix nonetype errors when the visit of a session does not exist so that proper logging is done (#824) * fix some errors when RepetitionTime is not available in JSON file (#825) * Add capability to download file from s3 (#826) * add capability to download file from s3 * fix flake8 error * Upload to S3: support object name starting with s3://bucket_name/ for upload (#827) * add capability to download file from s3 * fix flake8 error * add ability to remove s3://bucketname/ from the object name before upload * fix database class pselect documentation for the return type (#828) * map scan type to scan type ID when scan type provided to run_nifti_insertion.pl (#829) * modify permission of script run_push_imaging_files_to_s3_pipeline.py to make it executable (#830) * skip violation if not found on filesystem since it means the scan has been rerun (#831) * update VERSION file (#832) * do not push files to S3 when their path in the DB is already an S3 URL (#833) * fix violation files path when checking if the files are on the filesystem before adding them to the list of files to push to S3 (#834) * Merge 24.0 release into 24.1 release (#836) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix conflict * fix version * fix version Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * fix check if file already inserted in DB (#845) * Fix logic of determining file run number when previously inserted files are already pushed to S3 and not on filesystem anymore (#846) * fix bug * fix listing of filenames * comment new function * update version file (#847) * Chunk creation subprocess failure check (#848) * Chunk creation subprocess failure check Raise error when the chunk creation subprocess fails. Fix #843 * Update python/lib/physiological.py Print actual error message Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Revert chunk_pb2.py changes (#849) * remove prints in nifti_insertion_pipeline.py (#851) * fix permissoin denied upon deletion of tmp dir (#853) * update to next bug fix relesae (#854) * fix duplicated protocols error when same scan type returned (#856) * Add missing exit codes on the python's side (#857) * add some missing exit codes * add some missing exit codes * add ignore case to regex (#859) * add download from S3 and reupload if file provided to run_nifti-insertion was an S3 URL (#860) * fix intended for bug when no acq time available (#861) * fix bug for intended for when getting the list of files needed IntendedFor (#862) * fix paths when there are not / at the end of the Config (#866) Tested on sandbox with Config `data_dir` = `/data/loris/data` * fix NoneType error /opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_archive_loader_pipeline.py, line 346, in _add_intended_for_to_fieldmap_json_files (#867) Tested on HBCD sandbox on the dataset that caused the issue. * Properly update `mri_upload` 'Inserting' column when different sections of the pipeline are run (#868) * update mri_upload to inserting=0 when push to s3 pipeline is finished * fix Inserting flag being properly set when pipeline is running * update version file to 24.1.6 (#870) * Add download from S3 for fmap already pushed to S3 that needs to have IntendedFor written in them (#874) * add download from S3 for fmap that needs to have IntendedFor written in them * add print * update version to 24.1.7 (#876) * Merge 24.0.4 into 24.1 release (#878) * Reload the mri_upload dictionary before checking if a tarchive has been validated (#783) * reload mri_upload object * remove debugging exit and print statements * fix minor bugs when dealing with scans.tsv files (#774) * fix regex search for excluded series description patterns (#786) * fix_return_statement_of_create_imaging_upload_dict_from_upload_id_function (#787) * [dcm2bids] Insert into MRICandidateErrors if there is a Candidate PatientName mismatch (#790) * insert into MRICandidateErrors when candidate mismatch or pname not matching between DICOMs and NIfTI * remove exit * fix table name to MRICandidateErrors instead of MriCandidateErrors as apparently, it makes a difference on MariaDB/Linux VMs while it just worked on local install on Mac... (#793) * Set DICOM dates to undef if the date does not follow proper DICOM standard (#794) * set date to undef if it does not follow proper DICOM formats * fix all dates set to NULL * Installation and pet fixes (#818) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> * Update VERSION file for next bug fix release * Project, event validation and protobuf update (#823) * Project, event validation and protobuf update * Site and project search when creating candidate * missing import * correct pscid search * Events validation * flake rules update * review, and NULL value filtered out for site and project * flake * flake Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * DICOM Archive broken archive link fix (#872) * update version to 24.0.4 (#877) Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> * add a delete_file function for S3 (#881) * MATLAB 7.3 HDF5 Workaround (#880) * Added workaround for HDF5 files * Added correct file path confirmation * Duplicated code clean-up * Revised conditional logic * Fix nii insertion mri protocol group id bug (#883) * fix no attribute 'mri_protocol_group_id' error when scan type is specified * fix bug * refactor changes * fix bug * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * fix violation summary key unavailable * Do not register multiple entries of the same violation in `mri_violations_log` and `mri_protocol_violated_scans` (#884) * keep track of additional event data (24.1) (#873) * keep track of additional event data * py eeg/physiological * Copy .set file before updating paths (#886) * change version to 24.1.8 (#887) * fix incomplete format for query to files table for EchoNumber field (#889) * Add a check before inserting in mri_candidate_errors.py (#891) * add a check before inserting in mri_candidate_errors.py * use TE instead of imageType * fix gzip_nifti empty (#898) * Fix bug when scan type is provided to nifti_insertion.py script (#899) * fix NIfTI insertion scan type name attribute not found * fix no scan type id * fix nifti insertion when scan type is not provided (#901) * change Task to Property prefix for additional events (#900) * add image_type to mpvs (#902) * Fix run numbering logic to use SessionID and not TarchiveID to figure out the next number of a run (#903) * fix run number logic to use SessionID instead of TarchiveID to determine run number * fix bug to query the files for the sessionID instead of hte tarchiveid * fix doc * update version to 24.1.9 (#904) * get SessioNID from the session_obj instead of the tarchive obj (#909) * Update VERSION to 24.1.10 (#910) --------- Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mail.mcgill.ca> Co-authored-by: Laetitia Fesselier <laetitia.fesselier@mcgill.ca> Co-authored-by: regisoc <regis.ongaro-carcy@mcin.ca> Co-authored-by: regis <regisoc@users.noreply.github.com> Co-authored-by: jeffersoncasimir <15801528+jeffersoncasimir@users.noreply.github.com> Co-authored-by: Zaliqa <zaliqa.rosli@mcin.ca>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This modifies the
upload_filefunction ofAwsS3python class to be able to handle objects starting withs3://bucket_name/the same way it is handled bydownload_filefunction of the same class.