Kevin branch (#36)

* simplified loader/writer specification in IO ops

* updated imports in pipeline

* improved ops repr

* minor fixes in IO and segmentation

* matching roi names in StructureSet.to_segmentation is now case-insensitive

* fixed segmentation method call in some functional ops

* fixed typo in pipeline warning message

* StructureSet now correctly handles ROIs with no contour data

* fixed error when trying to detect CSV delimiter in MetadataWriter

* MetadataWriter can now remove existing files from previous runs

* MetadataWriter can now remove existing files from previous runs

* Segmentation object now correctly handles indexing

* fixed segmentation mask handling in image statistics computation

* removed old structure set code

* removed old structure set code

* fixed label handling in image statistics computation

* fixed label handling in image statistics computation

* [In Progress] Adding documentation to ops.py

* added support for dynamic path specification in writers

* image CSV loader now accepts pandas DataFrames

* fixed indentation error

* fixed index column handling in ImageCSVLoader

* Revert "fixed index column handling in ImageCSVLoader"

This reverts commit fab58d2.

* fixed index column handling in ImageCSVLoader

* fixed index column handling in ImageCSVLoader

* ImageCSVLoader now returns correct keys

* ImageCSVLoader now correctly handles globbing in paths

* more informative exception handling in pipeline

* pipeline can now warn on error instead of raising exception

* [Docs] Cleaned up extra whitespace in ops.py

* Added convenience method to get all ops in a pipeline

* Fix empty array check in structure set conversion

* Fix segmentation to label image conversion

* Update README.md

* Fix spurious mixin in ops

* Fix type error when passing Numpy array to rotate

* [Bugfix] Fix slice number rounding when generating a mask from RTSTRUCT

* Fix crop size issue in crop_to_bounding_box

* Make imports easier

* Improved handling of regular expressions in structure set

* Fix slice matching issue when generating binary mask from RTStruct

* Fix binary mask generation from RTStruct when missing labels are present

* little updates

* added seg.nrrd compatibility

* supports RTSTRUCT processing without roi_names

* Fixed contour (RTSTRUCT) handling to rasterize multiple contours better.

* Fixed contour (RTSTRUCT) handling to rasterize multiple contours better.

* fixed structureset.py:148

* RT dose pipeline completed and tested. Error in segmentation not resolved

* PET readability added

* PET readability added

* Head-Neck-PET-CT pipeline + `read_dicom_auto`

* final pipeline working for doses and PET

* corrected the view

* reading in sitk format from beginning

* Completed PET overlay

* Modified RT dose, pipeline working for PET_CT quebec dataset

* Added DataGraph, now fetching subset of dataset is supported using graph query

* added crawl.py

* Introduced changes in DataGraph. Made the pipeline fully general. Made wrapper classes

* Rectified some bugs, added reference to rtplan in the crawler

* Now missing RTDOSE references are completed by RTPLAN

* small change

* Fixed dataset.csv writing issues

* major refactoring

* more refactoring

* Update radcure_simple.py

* Update loaders.py

* Update pet.py

* Delete .DS_Store

* Delete .DS_Store

* Delete .DS_Store

* Delete .DS_Store

* Delete .DS_Store

* Delete .DS_Store

* Delete .DS_Store

* ignore .DS_Store

* Update README.md

* quick README under ops

* tcia_sample

* example bash script with path parsing, fixed reference_frame

* housekeeping

* fixed dataset.csv

* sanity changes before purging dev-20200414

* Added test autopipeline and modalities, solved some autopipeline bugs, read_dicom_series and pet now supports series_id


Former-commit-id: 03f9957

* PT/RTDOSE metadata to csv


Former-commit-id: 08616a5

* fixed some bugs in autopipeline.py


Former-commit-id: e05ea38

* now the pipeline saves on exit


Former-commit-id: 9cd421d

* deleted data


Former-commit-id: d3e6b6a

* now checks for existing subject id


Former-commit-id: bf4de75

* uncommented one line pytest


Former-commit-id: 3dc8cd3

* uncommented one line pytest


Former-commit-id: 2b9d6d4

* self.existing, dataset.csv fixed (#10)

* Added test autopipeline and modalities, solved some autopipeline bugs, read_dicom_series and pet now supports series_id

* PT/RTDOSE metadata to csv

* fixed some bugs in autopipeline.py

* now the pipeline saves on exit

* deleted data

* now checks for existing subject id

* uncommented one line pytest

* uncommented one line pytest

Co-authored-by: Vishwesh <vishweshramanathan@gmail.com>

* Added dataset class which can load from nrrds or directly from the dataset and convert to pytorch dataset


Former-commit-id: 1cd8984

* Create build-ci.yml

* Update build-ci.yml

* Update requirements.txt

* bug fixes_1.0


Former-commit-id: d255674

* test and autopipe fixed


Former-commit-id: 9cc88b1

* bug fixes 2


Former-commit-id: e7f4b42

* bug fixes 2


Former-commit-id: 90732d8

* added visualizations and some more bug fixes


Former-commit-id: 735b26c

* Create manual-test.yml

* Update build-ci.yml

* Update manual-test.yml

* PR tests - macos/ubuntu failing (#13)

* Added test autopipeline and modalities, solved some autopipeline bugs, read_dicom_series and pet now supports series_id

* PT/RTDOSE metadata to csv

* fixed some bugs in autopipeline.py

* now the pipeline saves on exit

* deleted data

* now checks for existing subject id

* uncommented one line pytest

* uncommented one line pytest

* Added dataset class which can load from nrrds or directly from the dataset and convert to pytorch dataset

* bug fixes_1.0

* test and autopipe fixed

* bug fixes 2

* fixed pipeline tests

* clean tests

* added workflow

* yml

* yml

* matplotlib

* trying other patient to avoid memoryerror

* set roi_names to avoid memoryerror

* cave

* indents

* Update manual-test.yml

Co-authored-by: Vishwesh <vishweshramanathan@gmail.com>

* fixed bugs regarding multiple connections, saving of metadata and loading of metadata


Former-commit-id: 1528efc

* small bug fix


Former-commit-id: 7a27ee3

* added demo.py


Former-commit-id: b644919

* Ready for

* Create main.yml (#15)

* Changed dataset class returns


Former-commit-id: 3c6d7f5

* fix conflicts


Former-commit-id: caad95a

* fixed test autopipe


Former-commit-id: 63b9543

* merging new features (#16)

* Added test autopipeline and modalities, solved some autopipeline bugs, read_dicom_series and pet now supports series_id

* PT/RTDOSE metadata to csv

* fixed some bugs in autopipeline.py

* now the pipeline saves on exit

* deleted data

* now checks for existing subject id

* uncommented one line pytest

* uncommented one line pytest

* Added dataset class which can load from nrrds or directly from the dataset and convert to pytorch dataset

* bug fixes_1.0

* test and autopipe fixed

* bug fixes 2

* bug fixes 2

* added visualizations and some more bug fixes

* fixed bugs regarding multiple connections, saving of metadata and loading of metadata

* small bug fix

* added demo.py

* Changed dataset class returns

* fix conflicts

* fixed test autopipe

Co-authored-by: Vishwesh <vishweshramanathan@gmail.com>

* fix path backslash issues


Former-commit-id: d81a74d

* fix path backslashes (#17)

* Added test autopipeline and modalities, solved some autopipeline bugs, read_dicom_series and pet now supports series_id

* PT/RTDOSE metadata to csv

* fixed some bugs in autopipeline.py

* now the pipeline saves on exit

* deleted data

* now checks for existing subject id

* uncommented one line pytest

* uncommented one line pytest

* Added dataset class which can load from nrrds or directly from the dataset and convert to pytorch dataset

* bug fixes_1.0

* test and autopipe fixed

* bug fixes 2

* bug fixes 2

* added visualizations and some more bug fixes

* fixed bugs regarding multiple connections, saving of metadata and loading of metadata

* small bug fix

* added demo.py

* Changed dataset class returns

* fix conflicts

* fixed test autopipe

* fix path backslash issues

Co-authored-by: Vishwesh <vishweshramanathan@gmail.com>

* Update main.yml

* Update main.yml

* Update README.md

* Update main.yml (#18)

* Update main.yml

* Update requirements.txt

* Update main.yml

* Update main.yml

* build binary/dist

* removed linter

* Update setup.py

* Update README.md

* Update README.md (#19)

* Update README.md

* added tests for Dataset class


Former-commit-id: 8d345fb

* added tests for Dataset class


Former-commit-id: 7747a81

* Create LICENSE (#20)

* Create LICENSE

* Update setup.py

* Seg.nrrd quick fix

* Minor bug fixes


Former-commit-id: f18aa1b

* test fix


Former-commit-id: 2fad37e

* Added demo


Former-commit-id: 48ea411

* Update setup.py (#23)



Former-commit-id: 9e24716

* updated README


Former-commit-id: 0f04145

* Update README.md (#24)



Former-commit-id: 9aaf51b

* preliminary MRI functionality (MR-RTSTRUCT pairs)


Former-commit-id: 3c01806

* Skim2257 quick fix (#26)

* Updated crawler to force String on all meta fields

* Update setup.py

Former-commit-id: 5e93381

* first commit


Former-commit-id: 971e8d6

* removed test files, changed gitignore


Former-commit-id: ff42f8a

* changed file directory structure for imageautooutput


Former-commit-id: c2d8c68

* split mask up into each contour


Former-commit-id: 7a8285c

* change kwargs in put for basesubjectwriter


Former-commit-id: a9f093c

* still kinda failing...


Former-commit-id: 358df05

* brought back basesubjectwriter


Former-commit-id: 9a8ff65

* .imgtools directory


Former-commit-id: 455c027

* changed absolute paths to relative paths


Former-commit-id: e1e5d0d

* changed os.path.join to pathlib.Path.as_posix()


Former-commit-id: 042aa1f

* removed unused cv2 import


Former-commit-id: 9d5bc56

* removed cv2 import


Former-commit-id: 36aab93

* appened is deprecated, changed to concat


Former-commit-id: f6d997b

* debug print


Former-commit-id: 5a6caa5

* removed debug print


Former-commit-id: 49c575a

* added sparse mask class and generating function


Former-commit-id: 8dab6bc

* testing out sparse mask


Former-commit-id: 19b0ad5

* funky NaN problem


Former-commit-id: 7e96504

* commented sparse mask


Former-commit-id: 915d769

* overwrite all subjects


Former-commit-id: f0ebe6a

* space


Former-commit-id: eb30814

* overwrite false


Former-commit-id: ff3a115

* metadata stuffs


Former-commit-id: c236408

* metadata in dataset.csv


Former-commit-id: ae09827

* added modalities, num rtstruct and pixel size to metadata


Former-commit-id: 51634e6

* metadata bugfix


Former-commit-id: 0042f14

* a


Former-commit-id: f43b45e

* fixed wrong variable names for metadata stuff


Former-commit-id: 67449ca

* fixed pathlib float error


Former-commit-id: aa36b2d

* relative paths and output folder paths for dataset.csv


Former-commit-id: 2ce5ade

* put metadata stuff into a util file


Former-commit-id: 69f4ee7

* deal with empty metadata


Former-commit-id: 31d0411

* messing around with sparse mask


Former-commit-id: a66925c

* tried to save sparse mask, did some stuff with nnunet output format


Former-commit-id: 6b54305

* compliant with nnunet directory structure


Former-commit-id: 1407e1d

* CLI Interface, argparse moved to utils


Former-commit-id: 6bf5f4f

* fixed formatting problems with folder names


Former-commit-id: 437db52

* train test split


Former-commit-id: 15bab7b

* train size and random state optional


Former-commit-id: 3ac4016

* merge conflicts


Former-commit-id: f96833d

* changed warnings to not interrupt


Former-commit-id: 154819e

* changed to warnings.warn for generate_sparse_mask


Former-commit-id: 1aec56d

* merge


Former-commit-id: 29bc242

* resolving conflicts?


Former-commit-id: 0d16b02

* args


Former-commit-id: f4f6d2d

* changes for roi names as a dict


Former-commit-id: 0479cb8

* added regex dictionary option for non nnunet runs


Former-commit-id: eeae5b7

* sparse mask global labelling for contour name: index


Former-commit-id: d65c6d8

* got rid of file_name_convention stuff


Former-commit-id: effac09

* conflicts resolved


Former-commit-id: ab1e87b

* yaml thing


Former-commit-id: 23b76e4

* added list capabilities for the roi names dictionary


Former-commit-id: 2293242

* dataset.json for nnunet


Former-commit-id: 156553f

* CLI "autonew"


Former-commit-id: c8351e4

* changed all mutable defaults to None


Former-commit-id: 41630cd

* moved autotest changes to autopipeline and addede a few CLI args


Former-commit-id: d8ff8d7

* getting ready for merge to live


Former-commit-id: a786724

* test_components, test_modalities works with new AutoPipeline


Former-commit-id: e51fe82

* overwrite changes and error fix for nan paths again


Former-commit-id: b138fe2

* fixed if statement


Former-commit-id: c6a5f4d

* joblib parallel


Former-commit-id: 7a47740

* warnings for missing patients


Former-commit-id: abc0428

* summary messages


Former-commit-id: 6729aa5

* updated, passing tests. Updated version to 0.4


Former-commit-id: 81d6795

* update test


Former-commit-id: 7bc3588

* yaml path cli


Former-commit-id: 67ef10d

* yaml error check


Former-commit-id: fdeabd1

* pandas error


Former-commit-id: 4745da2

* Fixed read_dicom_auto


Former-commit-id: a9a297e

* skips series check if seris is None


Former-commit-id: eb8fb65

* updated readme to reflect v0.4 changes


Former-commit-id: d355cf5

* updated readme


Former-commit-id: 84d607f

* minor change


Former-commit-id: 6a02036

* remove .idea


Former-commit-id: 8bd0a0f

* remote .idea


Former-commit-id: cf18c39

* git ignore


Former-commit-id: 7d5b46a

* refactor nnunetutils to nnunet


Former-commit-id: d996869

* updated versions

Former-commit-id: 41c8c88

Co-authored-by: przebieglykaziu <mkazmierski.poznan@gmail.com>
Co-authored-by: Minoru Nakano <minoru.nakano@gmail.com>
Co-authored-by: Sejin Kim <sejinkim@uhnslurmbuildbox.uhnh4h.cluster>
Co-authored-by: Sejin Kim <sejinkim@node88.uhnh4h.cluster>
Co-authored-by: Sejin Kim <sejinkim@node40.uhnh4h.cluster>
Co-authored-by: Sejin Kim <sejinkim@node97.uhnh4h.cluster>
Co-authored-by: Sejin Kim <sejinkim@h4huhnlogin2.uhnh4h.cluster>
Co-authored-by: Vishwesh Ramanathan <vishwesh@Vishweshs-Air.uhn.ca>
Co-authored-by: Sejin Kim <sejinkim@node38.uhnh4h.cluster>
Co-authored-by: Vishwesh Ramanathan <ramanav@node49.uhnh4h.cluster>
Co-authored-by: Vishwesh Ramanathan <ramanav@node53.uhnh4h.cluster>
Co-authored-by: Vishwesh <vishweshramanathan@gmail.com>
Co-authored-by: Benjamin Haibe-Kains <bhaibeka@bhaibek1.uhn.ca>
Co-authored-by: Sejin Kim <sejinkim@node89.uhnh4h.cluster>
Co-authored-by: Vishwesh Ramanathan <vishwesh@Vishweshs-MacBook-Air.local>
Co-authored-by: fishingguy456 <qukevin99@gmail.com>
Co-authored-by: Kevin Qu <kqu@uhnslurmbuildbox.uhnh4h.cluster>
Co-authored-by: Kevin Qu <kqu@node90.uhnh4h.cluster>