This repository has been archived by the owner on Nov 16, 2023. It is now read-only.
select contiguous data splits for val and train #200
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
squassina
commented
Feb 12, 2020
squassina
commented
Feb 12, 2020
squassina
commented
Feb 12, 2020
squassina
commented
Feb 12, 2020
examples/interpretation/notebooks/F3_block_training_and_evaluation_local.ipynb
Outdated
Show resolved
Hide resolved
squassina
commented
Feb 12, 2020
examples/interpretation/notebooks/HRNet_Penobscot_demo_notebook.ipynb
Outdated
Show resolved
Hide resolved
squassina
commented
Feb 12, 2020
squassina
commented
Feb 12, 2020
squassina
commented
Feb 12, 2020
yalaudah
reviewed
Feb 13, 2020
scripts/prepare_dutchf3.py
Outdated
Comment on lines
23
to
33
| loader_type + "_train_val.txt"), "w") | ||
| file_object.write("\n".join(train_list + test_list)) | ||
| file_object.close() | ||
| file_object = open(path.join(splits_path, loader_type + "_train.txt"), "w") | ||
| file_object = open(path.join(splits_path, | ||
| loader_type + "_train.txt"), "w") | ||
| file_object.write("\n".join(train_list)) | ||
| file_object.close() | ||
| file_object = open(path.join(splits_path, loader_type + "_val.txt"), "w") | ||
| file_object = open(path.join(splits_path, | ||
| loader_type + "_val.txt"), "w") | ||
| file_object.write("\n".join(test_list)) | ||
| file_object.close() |
There was a problem hiding this comment.
I assume test_list here is for validation right? Since the test is done on another volume? If this is correct, then please change this to val_list to avoid confusing the validation and testing splits
There was a problem hiding this comment.
All references to test is now val/validation
scripts/prepare_dutchf3.py
Outdated
| labels_path = _get_labels_path(data_dir) | ||
| logger.info(f"Loading {labels_path}") | ||
| labels = np.load(labels_path) | ||
| # labels_path = _get_labels_path(data_dir) |
There was a problem hiding this comment.
since this is no longer used, please delete it. Ideally, comments should not include code.
scripts/prepare_dutchf3.py
Outdated
Comment on lines
84
to
89
| train_iline_range, test_iline_range = _get_aline_range(iline, | ||
| per_val, | ||
| slice_steps) | ||
| train_i_list = ["i_" + str(i) for i in train_iline_range] | ||
| test_i_list = ["i_" + str(i) for i in test_iline_range] | ||
|
|
There was a problem hiding this comment.
_get_aline_range takes per_val as an input that specifies the percentage used for validation. So i would expect the output to betrain_iline_range, **val**_iline_range not test_iline_range. Its important we be careful with naming validation vs. testing to avoid confusion. So please replace any mention of test in this function with val
scripts/prepare_dutchf3.py
Outdated
Comment on lines
139
to
156
| train_iline_range, test_iline_range = _get_aline_range(iline, | ||
| per_val, | ||
| slice_steps) | ||
|
|
||
| # Xline sections | ||
| train_xline_range, test_xline_range = _get_aline_range(xline, per_val) | ||
| train_xline_range, test_xline_range = _get_aline_range(xline, | ||
| per_val, | ||
| slice_steps) | ||
|
|
||
| # Generate patches from sections | ||
| # Process inlines | ||
| horz_locations = range(0, xline - patch_size + 1, stride) | ||
| vert_locations = range(0, depth - patch_size + 1, stride) | ||
| logger.debug("Generating Inline patches") | ||
| logger.debug(horz_locations) | ||
| # Vertical locations is common to all patches processed | ||
| vert_locations = range(0, depth - patch_size, stride) | ||
| logger.debug(vert_locations) | ||
|
|
||
| # Process inlines | ||
| # iline = xline x depth | ||
| test_iline = math.floor(xline * per_val / 2) | ||
| logger.debug(test_iline) |
There was a problem hiding this comment.
Same comment here.. replace any mention of test with val
There was a problem hiding this comment.
All references to test is now val/validation
scripts/prepare_dutchf3.py
Outdated
Comment on lines
166
to
187
| horz_locations_train = range(test_iline, xline - patch_size, stride) | ||
| logger.debug("Generating Inline patches") | ||
| logger.debug("Generating Inline patches - Train") | ||
| logger.debug(horz_locations_train) | ||
| train_i_list = list(_i_extract_patches(train_iline_range, | ||
| horz_locations_train, | ||
| vert_locations)) | ||
| # test_iline - define size of the test set for the fist part | ||
| test_iline_range = list(test_iline_range) | ||
| test_i_list = list(_i_extract_patches(test_iline_range, horz_locations, vert_locations)) | ||
| train_i_list = list(_i_extract_patches(train_iline_range, horz_locations, vert_locations)) | ||
|
|
||
| # Process inlines - test - begining | ||
| logger.debug("Generating Inline patches - Test") | ||
| horz_locations_test_begin = range(0, test_iline, max(1,stride)) | ||
| test_i_list = list(_i_extract_patches(test_iline_range, | ||
| horz_locations_test_begin, | ||
| vert_locations)) | ||
| # Process inlines - test - end | ||
| horz_locations_test_end = range(xline - stride + 1, xline, max(1,stride)) | ||
| test_i_list += list(_i_extract_patches(test_iline_range, | ||
| horz_locations_test_end, | ||
| vert_locations)) | ||
|
|
There was a problem hiding this comment.
all references to test here should be val correct?
There was a problem hiding this comment.
All references to test is now val/validation
maxkazmsft
suggested changes
Feb 13, 2020
yalaudah
reviewed
Feb 13, 2020
scripts/prepare_dutchf3.py
Outdated
| logger.debug(vert_locations) | ||
|
|
||
| # Process inlines | ||
| # iline = xline x depth | ||
| val_iline = math.floor(xline * per_val / 2) |
There was a problem hiding this comment.
why do we divide by 2 here? is it because the val sections are taken from both ends of the volume?
There was a problem hiding this comment.
yes, we split half to each edge of the volume.
scripts/prepare_dutchf3.py
Outdated
| logger.debug(val_xline) | ||
|
|
||
| # Process xlines - train | ||
| horz_locations_train = range(val_xline, iline - patch_size, stride) |
There was a problem hiding this comment.
not sure i'm following here. Why is the range from val_xline to iline - patch_size? <-- it might be easier to jump on a quick call to discuss this.
scripts/prepare_dutchf3.py
Outdated
| # create train and test splits: | ||
| train_list, test_list = train_test_split(list_train_val, test_size=fraction_validation, shuffle=True) | ||
| # create train and validation splits: | ||
| train_list, val_list = train_val_split(list_train_val, val_size=fraction_validation, shuffle=True) |
There was a problem hiding this comment.
this is a function imported from scikit-learn. it should be train_test_split not train_val_split.. otherwise this would fail.
There was a problem hiding this comment.
same goes for val_size.. it should be test_size
There was a problem hiding this comment.
I didn't catch this as an error because it is inside the split_alaudah_et_al_19 method. But I returned it to the original value.
maxkazmsft
previously approved these changes
Feb 13, 2020
yalaudah
changed the title
maxkazmsft
approved these changes
Feb 18, 2020
This was referenced Feb 19, 2020
|
Issue #233 should also address this PR as well. |
maxkazmsft
pushed a commit
that referenced
this pull request
Jun 9, 2020
* select contiguous data splits for test and train * changed data-dir to data_dir as arg to prepare_dutchf3.py * update script with new required parameter label_file * ignoring split_alaudah_et_al_19 as it is not updated * changed TEST to VALIDATION for clarity in the code * included job to run scripts unit test * Fix val/train split and add tests * adjust to consider the whole horz_lines * update environment - gitpython version
maxkazmsft
added a commit
that referenced
this pull request
Jun 15, 2020
* Adding content to interpretation README (#171) * added sharat, weehyong to authors * adding a download script for Dutch F3 dataset * Adding script instructions for dutch f3 * Update README.md prepare scripts expect root level directory for dutch f3 dataset. (it is downloaded into $dir/data by the script) * Adding readme text for the notebooks and checking if config is correctly setup * fixing prepare script example * Adding more content to interpretation README * Update README.md * Update HRNet_Penobscot_demo_notebook.ipynb Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * Updates to prepare dutchf3 (#185) * updating patch to patch_size when we are using it as an integer * modifying the range function in the prepare_dutchf3 script to get all of our data * updating path to logging.config so the script can locate it * manually reverting back log path to troubleshoot build tests * updating patch to patch_size for testing on preprocessing scripts * updating patch to patch_size where applicable in ablation.sh * reverting back changes on ablation.sh to validate build pass * update patch to patch_size in ablation.sh (#191) Co-authored-by: Sharat Chikkerur <sharat.chikkerur@gmail.com> * TestLoader's support for custom paths (#196) * Add testloader support for custom paths. * Add test * added file name workaround for Train*Loader classes * adding comments and clean up * Remove legacy code. * Remove parameters that dont exist in init() from documentation. * Add unit tests for data loaders in dutchf3 * moved unit tests Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * select contiguous data splits for val and train (#200) * select contiguous data splits for test and train * changed data-dir to data_dir as arg to prepare_dutchf3.py * update script with new required parameter label_file * ignoring split_alaudah_et_al_19 as it is not updated * changed TEST to VALIDATION for clarity in the code * included job to run scripts unit test * Fix val/train split and add tests * adjust to consider the whole horz_lines * update environment - gitpython version * Segy Converter Utility (#199) * Add convert_segy utility script and related notebooks * add segy files to .gitignore * readability update * Create methods for normalizing and clipping separately. * Add comment * update file paths * cleanup tests and terminology for the normalization/clipping code * update notes to provide more context for using the script * Add tests for clipping. * Update comments * added Microsoft copyright * Update root README * Add a flag to turn on clipping in dataprep script. * Remove hard coded values and fix _filder_data method. * Fix some minor issues pointed out on comments. * Remove unused lib. * Rename notebooks to impose order; set env; move all def funtions into utils; improve comments in notebooks; and include code example to run prepare_dutchf3.py * Label missing data with 255. * Remove cell with --help command. * Add notebooks to test pipeline. * grammer edits * update notebook output and utils naming * fix output dir error and cleanup notebook * fix yaml indent error in notebooks_build.yml * fix merge issues and job name errors * debugging the build pipeline * combine notebook tests for segy converter since they are dependent on each other Co-authored-by: Geisa Faustino <32823639+GeisaFaustino@users.noreply.github.com> * Azureml train pipeline (#195) * initial add of azure ml pipeline * update references and dependencies * fix integration tests * remove incomplete tests * add azureml requirements.txt for dutchf3 local patch and update pipeline config * add empty __init__.py to cv_lib dutchf3 * Get train,py to run in pipeline * allow output dir in train.py * Clean up README and __init__ * only pass output if available and use input dir for output in train.py * update comment in train.py * updating azureml_requirements to only pull from /master * removing windows guidance in azureml_pipelines/README.md * adding .env.example * adding azureml config example * updating documentation in azureml_pipelines README.md * updating main README.md to refer to AML guidance documentation * updating AML README.md to include additional guidance to cancel runs * adding documentation on AzureML pipelines in the AML README.me * adding files needed section for AML training run * including hyperlink in format poiniting to additional detail on Azure Machine Learning pipeslines in AML README.md * removing the mention of VSCode in the AML README.md * fixing typo * modifying config to pipeline configuration in README.md * fixing typo in README.md * adding documentation on how to create a blob container and copy data onto it * adding documentation on blob storage guidance * adding guidance on how to get the subscription id * adding guidance to activate environment and then run the kick off train pipeline from ROOT * adding ability to pass in experiement name and different pipeline configuration to kickoff_train_pipeline.py * adding Microsoft Corporation Copyright to kickoff_train_pipeline.py * fixing format in README.md * adding trouble shooting section in README.md for connection to subscription * updating troubleshooting title * adding guidance on how to download the config.json from the Azure Portal in the README.md * adding additional guidance and information on AzureML compute targets and naming conventions * changing the configuation file example to only include the train step that is currently supported * updating config to pipeline configuration when applicable * adding link to Microsoft docs for additional information on pipeline steps * updated AML test build definitions * updated AML test build definitions * adding job to aml_build.yml * updating example config for testing * modifying the test_train_pipeline.py to have appropriate number of pipeline steps and other required modifications * updating AML_pipeline_tests in aml_build.yml to consume environment variables * updating scriptType, sciptLocation, and inlineScript in aml_build.yml * trivial commit to re-trigger broken build pipelines * fix to aml yml build to use env vars for secrets and everything else * another yml fix * another yml fix * reverting structure format of jobs for aml_build pipeline tests * updating path to test_train_pipeline.py * aml_pipeline_tests timed out, extending timeoutInMinutes from 10 to 40 * adding additional pytest * adding az login * updating variables in aml pipeline tests Co-authored-by: Anna Zietlow <annamzietlow@gmail.com> Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * moved contrib contributions around from CSE * fixed dataloader tests - updated them to work with new code from staging branch * segyconverter notebooks and tests run and pass; updated documentation * added test job for segy converter notebooks * removed AML training pipeline from this release * fixed training model tolerance precision in the tests - wasn't working * fixed train.py build issues after the merge * addressed PR comments * fixed bug in check_performance Co-authored-by: Sharat Chikkerur <sharat.chikkerur@microsoft.com> Co-authored-by: kirasoderstrom <kirasoderstrom@gmail.com> Co-authored-by: Sharat Chikkerur <sharat.chikkerur@gmail.com> Co-authored-by: Geisa Faustino <32823639+GeisaFaustino@users.noreply.github.com> Co-authored-by: Ricardo Squassina Lee <8495707+squassina@users.noreply.github.com> Co-authored-by: Michael Zawacki <mikezawacki@hotmail.com> Co-authored-by: Anna Zietlow <annamzietlow@gmail.com>
maxkazmsft
added a commit
that referenced
this pull request
Jul 7, 2020
* cleaning up files which are no longer needed * fixes after removing forking workflow (#322) * PR to resolve merge issues * updated main build as well * added ability to read in git branch name directly * manually updated the other files * fixed number of classes for main build tests (#327) * fixed number of classes for main build tests * corrected DATASET.ROOT in builds * added dev build script * Fixes for development inside the docker container (#335) * Fix the mound command for the HRNet pretrained model in the docker readme * Properly catch InvalidGitRepository exception * make repo paths consistent with non-docker runs -- this way configs paths do not need to be changed * Properly catch InvalidGitRepository exception in train.py * Readme update (#337) * README updates * Removing user specific path from config Authored-by: Fatemeh Zamanian <Fatemeh.Zamanian@microsoft.com> * Fixing #324 and #325 (#338) * update colormap to a non-discrete one -- fixes #324 * fix mask_to_disk to normalize by n_classes * changes to test.py * Updating data.py * bug fix * increased timeout time for main_build * retrigger build * retrigger the build * increase timeout * fixes 318 (#339) * finished 318 * increased checkerboard test timeout * fix 333 (#340) * added label correction to train gradient * changing the gradient data generator to take inline/crossline argument conssistent with the patchloader * changing variable name to be more descriptive Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * bug fix to model predictions (#345) * replace hrnet with seresnet in experiments - provides stable default model (#343) * PR to fix #342 (#347) * intermediate work for normalization * 1) normalize function runs based on global MIN and MAX 2) has a error handling for division by zero, np.finfo 3) decode_segmap normalizes the label/mask based on the n_calsses * global normalization added to test.py * increasing the threshold on timeout * trigger * revert * idk what happened * increase timeout * picking up global min and max * passing config to TrainPatchLoader to facilitate access to global min and max and other attr in low level functions, WIP * removed print statement * changed section loaders * updated test for min and max from config too * adde MIN and MAX to config * notebook modified for loaders * another dataloader in notebook * readme update * changed the default values for min max, updated the docstring for loaders, removed suppressed lines * debug * merging work from CSE team into main staging branch (#357) * Adding content to interpretation README (#171) * added sharat, weehyong to authors * adding a download script for Dutch F3 dataset * Adding script instructions for dutch f3 * Update README.md prepare scripts expect root level directory for dutch f3 dataset. (it is downloaded into $dir/data by the script) * Adding readme text for the notebooks and checking if config is correctly setup * fixing prepare script example * Adding more content to interpretation README * Update README.md * Update HRNet_Penobscot_demo_notebook.ipynb Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * Updates to prepare dutchf3 (#185) * updating patch to patch_size when we are using it as an integer * modifying the range function in the prepare_dutchf3 script to get all of our data * updating path to logging.config so the script can locate it * manually reverting back log path to troubleshoot build tests * updating patch to patch_size for testing on preprocessing scripts * updating patch to patch_size where applicable in ablation.sh * reverting back changes on ablation.sh to validate build pass * update patch to patch_size in ablation.sh (#191) Co-authored-by: Sharat Chikkerur <sharat.chikkerur@gmail.com> * TestLoader's support for custom paths (#196) * Add testloader support for custom paths. * Add test * added file name workaround for Train*Loader classes * adding comments and clean up * Remove legacy code. * Remove parameters that dont exist in init() from documentation. * Add unit tests for data loaders in dutchf3 * moved unit tests Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * select contiguous data splits for val and train (#200) * select contiguous data splits for test and train * changed data-dir to data_dir as arg to prepare_dutchf3.py * update script with new required parameter label_file * ignoring split_alaudah_et_al_19 as it is not updated * changed TEST to VALIDATION for clarity in the code * included job to run scripts unit test * Fix val/train split and add tests * adjust to consider the whole horz_lines * update environment - gitpython version * Segy Converter Utility (#199) * Add convert_segy utility script and related notebooks * add segy files to .gitignore * readability update * Create methods for normalizing and clipping separately. * Add comment * update file paths * cleanup tests and terminology for the normalization/clipping code * update notes to provide more context for using the script * Add tests for clipping. * Update comments * added Microsoft copyright * Update root README * Add a flag to turn on clipping in dataprep script. * Remove hard coded values and fix _filder_data method. * Fix some minor issues pointed out on comments. * Remove unused lib. * Rename notebooks to impose order; set env; move all def funtions into utils; improve comments in notebooks; and include code example to run prepare_dutchf3.py * Label missing data with 255. * Remove cell with --help command. * Add notebooks to test pipeline. * grammer edits * update notebook output and utils naming * fix output dir error and cleanup notebook * fix yaml indent error in notebooks_build.yml * fix merge issues and job name errors * debugging the build pipeline * combine notebook tests for segy converter since they are dependent on each other Co-authored-by: Geisa Faustino <32823639+GeisaFaustino@users.noreply.github.com> * Azureml train pipeline (#195) * initial add of azure ml pipeline * update references and dependencies * fix integration tests * remove incomplete tests * add azureml requirements.txt for dutchf3 local patch and update pipeline config * add empty __init__.py to cv_lib dutchf3 * Get train,py to run in pipeline * allow output dir in train.py * Clean up README and __init__ * only pass output if available and use input dir for output in train.py * update comment in train.py * updating azureml_requirements to only pull from /master * removing windows guidance in azureml_pipelines/README.md * adding .env.example * adding azureml config example * updating documentation in azureml_pipelines README.md * updating main README.md to refer to AML guidance documentation * updating AML README.md to include additional guidance to cancel runs * adding documentation on AzureML pipelines in the AML README.me * adding files needed section for AML training run * including hyperlink in format poiniting to additional detail on Azure Machine Learning pipeslines in AML README.md * removing the mention of VSCode in the AML README.md * fixing typo * modifying config to pipeline configuration in README.md * fixing typo in README.md * adding documentation on how to create a blob container and copy data onto it * adding documentation on blob storage guidance * adding guidance on how to get the subscription id * adding guidance to activate environment and then run the kick off train pipeline from ROOT * adding ability to pass in experiement name and different pipeline configuration to kickoff_train_pipeline.py * adding Microsoft Corporation Copyright to kickoff_train_pipeline.py * fixing format in README.md * adding trouble shooting section in README.md for connection to subscription * updating troubleshooting title * adding guidance on how to download the config.json from the Azure Portal in the README.md * adding additional guidance and information on AzureML compute targets and naming conventions * changing the configuation file example to only include the train step that is currently supported * updating config to pipeline configuration when applicable * adding link to Microsoft docs for additional information on pipeline steps * updated AML test build definitions * updated AML test build definitions * adding job to aml_build.yml * updating example config for testing * modifying the test_train_pipeline.py to have appropriate number of pipeline steps and other required modifications * updating AML_pipeline_tests in aml_build.yml to consume environment variables * updating scriptType, sciptLocation, and inlineScript in aml_build.yml * trivial commit to re-trigger broken build pipelines * fix to aml yml build to use env vars for secrets and everything else * another yml fix * another yml fix * reverting structure format of jobs for aml_build pipeline tests * updating path to test_train_pipeline.py * aml_pipeline_tests timed out, extending timeoutInMinutes from 10 to 40 * adding additional pytest * adding az login * updating variables in aml pipeline tests Co-authored-by: Anna Zietlow <annamzietlow@gmail.com> Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * moved contrib contributions around from CSE * fixed dataloader tests - updated them to work with new code from staging branch * segyconverter notebooks and tests run and pass; updated documentation * added test job for segy converter notebooks * removed AML training pipeline from this release * fixed training model tolerance precision in the tests - wasn't working * fixed train.py build issues after the merge * addressed PR comments * fixed bug in check_performance Co-authored-by: Sharat Chikkerur <sharat.chikkerur@microsoft.com> Co-authored-by: kirasoderstrom <kirasoderstrom@gmail.com> Co-authored-by: Sharat Chikkerur <sharat.chikkerur@gmail.com> Co-authored-by: Geisa Faustino <32823639+GeisaFaustino@users.noreply.github.com> Co-authored-by: Ricardo Squassina Lee <8495707+squassina@users.noreply.github.com> Co-authored-by: Michael Zawacki <mikezawacki@hotmail.com> Co-authored-by: Anna Zietlow <annamzietlow@gmail.com> * make tests simpler (#368) * removed Dutch F3 job from main_build * fixed a bug in data subset in debug mode * modified epoch numbers to pass the performance checks, checkedout check_performance from Max's branch * modified get_data_for_builds.sh to set up checkerboard data for smaller size, minor improvements on gen_checkerboard * send all the batches, disabled the performance checks for patch_deconvnet * added comment to enable tests for patch_deconvnet after debugging, renamed gen_checkerboard, added options to new arg per Max's suggestion * Replace HRNet with SEResNet model in the notebook (#362) * replaced HRNet with SEResNet model in the notebook * removed debugging cell info * fixed bug where resnet_unet model wasn't loading the pre-trained version in the notebook * fixed build VM problems * Multi-GPU training support (#359) * Data flow tests (#375) * renamed checkerboard job name * restructured default outputs from test.py to be dumped under output dir and not debug dir * test.py output re-org * removed outdated variable from check_performance.py * intermediate work * intermediate work * bunch of intermediate works * changing args for different trainings * final to run dev_build" * remove print statements * removed print statement * removed suppressed lines * added assertion error msg * added assertion error msg, one intential bug to test * testing a stupid bug * debug * omg * final * trigger build * fixed multi-GPU termination in train.py (#379) * PR to fix #371 and #372 (#380) * added learning rate to logs * changed epoch for patch_deconvnet, and enabled the tests * removed TODOs * changed tensorflow pinned version (#387) * changed tensorflow pinned version * trigger build * closes 385 (#389) * Fixing #259 by adding symmetric padding along depth direction (#386) * BYOD Penobscot (#390) * minor updates to files * added penobscot conversion code * docker build test (#388) * added a new job to test bulding the docker, for now it is daisy-chained to the end * this is just a TEST * test * test * remove old image * debug * debug * test * debug * enabled all the jobs * quick fix * removing non-tagged iamges Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * added missing license headers and fixed formatting (#391) * added missing license headers and fixed formatting * some more license headers * updated documentation to close 354 and 381 (#392) * fix test.py and notebook issues (#394) * resolved conflicts for 0.2 release (#396) * V00.01.00003 release (#356) * cleaning up files which are no longer needed * fixes after removing forking workflow (#322) * PR to resolve merge issues * updated main build as well * added ability to read in git branch name directly * manually updated the other files * fixed number of classes for main build tests (#327) * fixed number of classes for main build tests * corrected DATASET.ROOT in builds * added dev build script * Fixes for development inside the docker container (#335) * Fix the mound command for the HRNet pretrained model in the docker readme * Properly catch InvalidGitRepository exception * make repo paths consistent with non-docker runs -- this way configs paths do not need to be changed * Properly catch InvalidGitRepository exception in train.py * Readme update (#337) * README updates * Removing user specific path from config Authored-by: Fatemeh Zamanian <Fatemeh.Zamanian@microsoft.com> * Fixing #324 and #325 (#338) * update colormap to a non-discrete one -- fixes #324 * fix mask_to_disk to normalize by n_classes * changes to test.py * Updating data.py * bug fix * increased timeout time for main_build * retrigger build * retrigger the build * increase timeout * fixes 318 (#339) * finished 318 * increased checkerboard test timeout * fix 333 (#340) * added label correction to train gradient * changing the gradient data generator to take inline/crossline argument conssistent with the patchloader * changing variable name to be more descriptive Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * bug fix to model predictions (#345) * replace hrnet with seresnet in experiments - provides stable default model (#343) Co-authored-by: yalaudah <yazeed.alaudah@microsoft.com> Co-authored-by: Fatemeh <fazamani@microsoft.com> * typos Co-authored-by: yalaudah <yazeed.alaudah@microsoft.com> Co-authored-by: Fatemeh <fazamani@microsoft.com> Co-authored-by: yalaudah <yazeed.alaudah@microsoft.com> Co-authored-by: Fatemeh <fazamani@microsoft.com> Co-authored-by: Sharat Chikkerur <sharat.chikkerur@microsoft.com> Co-authored-by: kirasoderstrom <kirasoderstrom@gmail.com> Co-authored-by: Sharat Chikkerur <sharat.chikkerur@gmail.com> Co-authored-by: Geisa Faustino <32823639+GeisaFaustino@users.noreply.github.com> Co-authored-by: Ricardo Squassina Lee <8495707+squassina@users.noreply.github.com> Co-authored-by: Michael Zawacki <mikezawacki@hotmail.com> Co-authored-by: Anna Zietlow <annamzietlow@gmail.com>
maxkazmsft
added a commit
that referenced
this pull request
Jul 22, 2020
* cleaning up files which are no longer needed * fixes after removing forking workflow (#322) * PR to resolve merge issues * updated main build as well * added ability to read in git branch name directly * manually updated the other files * fixed number of classes for main build tests (#327) * fixed number of classes for main build tests * corrected DATASET.ROOT in builds * added dev build script * Fixes for development inside the docker container (#335) * Fix the mound command for the HRNet pretrained model in the docker readme * Properly catch InvalidGitRepository exception * make repo paths consistent with non-docker runs -- this way configs paths do not need to be changed * Properly catch InvalidGitRepository exception in train.py * Readme update (#337) * README updates * Removing user specific path from config Authored-by: Fatemeh Zamanian <Fatemeh.Zamanian@microsoft.com> * Fixing #324 and #325 (#338) * update colormap to a non-discrete one -- fixes #324 * fix mask_to_disk to normalize by n_classes * changes to test.py * Updating data.py * bug fix * increased timeout time for main_build * retrigger build * retrigger the build * increase timeout * fixes 318 (#339) * finished 318 * increased checkerboard test timeout * fix 333 (#340) * added label correction to train gradient * changing the gradient data generator to take inline/crossline argument conssistent with the patchloader * changing variable name to be more descriptive Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * bug fix to model predictions (#345) * replace hrnet with seresnet in experiments - provides stable default model (#343) * PR to fix #342 (#347) * intermediate work for normalization * 1) normalize function runs based on global MIN and MAX 2) has a error handling for division by zero, np.finfo 3) decode_segmap normalizes the label/mask based on the n_calsses * global normalization added to test.py * increasing the threshold on timeout * trigger * revert * idk what happened * increase timeout * picking up global min and max * passing config to TrainPatchLoader to facilitate access to global min and max and other attr in low level functions, WIP * removed print statement * changed section loaders * updated test for min and max from config too * adde MIN and MAX to config * notebook modified for loaders * another dataloader in notebook * readme update * changed the default values for min max, updated the docstring for loaders, removed suppressed lines * debug * merging work from CSE team into main staging branch (#357) * Adding content to interpretation README (#171) * added sharat, weehyong to authors * adding a download script for Dutch F3 dataset * Adding script instructions for dutch f3 * Update README.md prepare scripts expect root level directory for dutch f3 dataset. (it is downloaded into $dir/data by the script) * Adding readme text for the notebooks and checking if config is correctly setup * fixing prepare script example * Adding more content to interpretation README * Update README.md * Update HRNet_Penobscot_demo_notebook.ipynb Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * Updates to prepare dutchf3 (#185) * updating patch to patch_size when we are using it as an integer * modifying the range function in the prepare_dutchf3 script to get all of our data * updating path to logging.config so the script can locate it * manually reverting back log path to troubleshoot build tests * updating patch to patch_size for testing on preprocessing scripts * updating patch to patch_size where applicable in ablation.sh * reverting back changes on ablation.sh to validate build pass * update patch to patch_size in ablation.sh (#191) Co-authored-by: Sharat Chikkerur <sharat.chikkerur@gmail.com> * TestLoader's support for custom paths (#196) * Add testloader support for custom paths. * Add test * added file name workaround for Train*Loader classes * adding comments and clean up * Remove legacy code. * Remove parameters that dont exist in init() from documentation. * Add unit tests for data loaders in dutchf3 * moved unit tests Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * select contiguous data splits for val and train (#200) * select contiguous data splits for test and train * changed data-dir to data_dir as arg to prepare_dutchf3.py * update script with new required parameter label_file * ignoring split_alaudah_et_al_19 as it is not updated * changed TEST to VALIDATION for clarity in the code * included job to run scripts unit test * Fix val/train split and add tests * adjust to consider the whole horz_lines * update environment - gitpython version * Segy Converter Utility (#199) * Add convert_segy utility script and related notebooks * add segy files to .gitignore * readability update * Create methods for normalizing and clipping separately. * Add comment * update file paths * cleanup tests and terminology for the normalization/clipping code * update notes to provide more context for using the script * Add tests for clipping. * Update comments * added Microsoft copyright * Update root README * Add a flag to turn on clipping in dataprep script. * Remove hard coded values and fix _filder_data method. * Fix some minor issues pointed out on comments. * Remove unused lib. * Rename notebooks to impose order; set env; move all def funtions into utils; improve comments in notebooks; and include code example to run prepare_dutchf3.py * Label missing data with 255. * Remove cell with --help command. * Add notebooks to test pipeline. * grammer edits * update notebook output and utils naming * fix output dir error and cleanup notebook * fix yaml indent error in notebooks_build.yml * fix merge issues and job name errors * debugging the build pipeline * combine notebook tests for segy converter since they are dependent on each other Co-authored-by: Geisa Faustino <32823639+GeisaFaustino@users.noreply.github.com> * Azureml train pipeline (#195) * initial add of azure ml pipeline * update references and dependencies * fix integration tests * remove incomplete tests * add azureml requirements.txt for dutchf3 local patch and update pipeline config * add empty __init__.py to cv_lib dutchf3 * Get train,py to run in pipeline * allow output dir in train.py * Clean up README and __init__ * only pass output if available and use input dir for output in train.py * update comment in train.py * updating azureml_requirements to only pull from /master * removing windows guidance in azureml_pipelines/README.md * adding .env.example * adding azureml config example * updating documentation in azureml_pipelines README.md * updating main README.md to refer to AML guidance documentation * updating AML README.md to include additional guidance to cancel runs * adding documentation on AzureML pipelines in the AML README.me * adding files needed section for AML training run * including hyperlink in format poiniting to additional detail on Azure Machine Learning pipeslines in AML README.md * removing the mention of VSCode in the AML README.md * fixing typo * modifying config to pipeline configuration in README.md * fixing typo in README.md * adding documentation on how to create a blob container and copy data onto it * adding documentation on blob storage guidance * adding guidance on how to get the subscription id * adding guidance to activate environment and then run the kick off train pipeline from ROOT * adding ability to pass in experiement name and different pipeline configuration to kickoff_train_pipeline.py * adding Microsoft Corporation Copyright to kickoff_train_pipeline.py * fixing format in README.md * adding trouble shooting section in README.md for connection to subscription * updating troubleshooting title * adding guidance on how to download the config.json from the Azure Portal in the README.md * adding additional guidance and information on AzureML compute targets and naming conventions * changing the configuation file example to only include the train step that is currently supported * updating config to pipeline configuration when applicable * adding link to Microsoft docs for additional information on pipeline steps * updated AML test build definitions * updated AML test build definitions * adding job to aml_build.yml * updating example config for testing * modifying the test_train_pipeline.py to have appropriate number of pipeline steps and other required modifications * updating AML_pipeline_tests in aml_build.yml to consume environment variables * updating scriptType, sciptLocation, and inlineScript in aml_build.yml * trivial commit to re-trigger broken build pipelines * fix to aml yml build to use env vars for secrets and everything else * another yml fix * another yml fix * reverting structure format of jobs for aml_build pipeline tests * updating path to test_train_pipeline.py * aml_pipeline_tests timed out, extending timeoutInMinutes from 10 to 40 * adding additional pytest * adding az login * updating variables in aml pipeline tests Co-authored-by: Anna Zietlow <annamzietlow@gmail.com> Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * moved contrib contributions around from CSE * fixed dataloader tests - updated them to work with new code from staging branch * segyconverter notebooks and tests run and pass; updated documentation * added test job for segy converter notebooks * removed AML training pipeline from this release * fixed training model tolerance precision in the tests - wasn't working * fixed train.py build issues after the merge * addressed PR comments * fixed bug in check_performance Co-authored-by: Sharat Chikkerur <sharat.chikkerur@microsoft.com> Co-authored-by: kirasoderstrom <kirasoderstrom@gmail.com> Co-authored-by: Sharat Chikkerur <sharat.chikkerur@gmail.com> Co-authored-by: Geisa Faustino <32823639+GeisaFaustino@users.noreply.github.com> Co-authored-by: Ricardo Squassina Lee <8495707+squassina@users.noreply.github.com> Co-authored-by: Michael Zawacki <mikezawacki@hotmail.com> Co-authored-by: Anna Zietlow <annamzietlow@gmail.com> * make tests simpler (#368) * removed Dutch F3 job from main_build * fixed a bug in data subset in debug mode * modified epoch numbers to pass the performance checks, checkedout check_performance from Max's branch * modified get_data_for_builds.sh to set up checkerboard data for smaller size, minor improvements on gen_checkerboard * send all the batches, disabled the performance checks for patch_deconvnet * added comment to enable tests for patch_deconvnet after debugging, renamed gen_checkerboard, added options to new arg per Max's suggestion * Replace HRNet with SEResNet model in the notebook (#362) * replaced HRNet with SEResNet model in the notebook * removed debugging cell info * fixed bug where resnet_unet model wasn't loading the pre-trained version in the notebook * fixed build VM problems * Multi-GPU training support (#359) * Data flow tests (#375) * renamed checkerboard job name * restructured default outputs from test.py to be dumped under output dir and not debug dir * test.py output re-org * removed outdated variable from check_performance.py * intermediate work * intermediate work * bunch of intermediate works * changing args for different trainings * final to run dev_build" * remove print statements * removed print statement * removed suppressed lines * added assertion error msg * added assertion error msg, one intential bug to test * testing a stupid bug * debug * omg * final * trigger build * fixed multi-GPU termination in train.py (#379) * PR to fix #371 and #372 (#380) * added learning rate to logs * changed epoch for patch_deconvnet, and enabled the tests * removed TODOs * changed tensorflow pinned version (#387) * changed tensorflow pinned version * trigger build * closes 385 (#389) * Fixing #259 by adding symmetric padding along depth direction (#386) * BYOD Penobscot (#390) * minor updates to files * added penobscot conversion code * docker build test (#388) * added a new job to test bulding the docker, for now it is daisy-chained to the end * this is just a TEST * test * test * remove old image * debug * debug * test * debug * enabled all the jobs * quick fix * removing non-tagged iamges Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * added missing license headers and fixed formatting (#391) * added missing license headers and fixed formatting * some more license headers * updated documentation to close 354 and 381 (#392) * fix test.py and notebook issues (#394) * resolved conflicts for 0.2 release (#396) * V00.01.00003 release (#356) * cleaning up files which are no longer needed * fixes after removing forking workflow (#322) * PR to resolve merge issues * updated main build as well * added ability to read in git branch name directly * manually updated the other files * fixed number of classes for main build tests (#327) * fixed number of classes for main build tests * corrected DATASET.ROOT in builds * added dev build script * Fixes for development inside the docker container (#335) * Fix the mound command for the HRNet pretrained model in the docker readme * Properly catch InvalidGitRepository exception * make repo paths consistent with non-docker runs -- this way configs paths do not need to be changed * Properly catch InvalidGitRepository exception in train.py * Readme update (#337) * README updates * Removing user specific path from config Authored-by: Fatemeh Zamanian <Fatemeh.Zamanian@microsoft.com> * Fixing #324 and #325 (#338) * update colormap to a non-discrete one -- fixes #324 * fix mask_to_disk to normalize by n_classes * changes to test.py * Updating data.py * bug fix * increased timeout time for main_build * retrigger build * retrigger the build * increase timeout * fixes 318 (#339) * finished 318 * increased checkerboard test timeout * fix 333 (#340) * added label correction to train gradient * changing the gradient data generator to take inline/crossline argument conssistent with the patchloader * changing variable name to be more descriptive Co-authored-by: maxkazmsft <maxkaz@microsoft.com> * bug fix to model predictions (#345) * replace hrnet with seresnet in experiments - provides stable default model (#343) Co-authored-by: yalaudah <yazeed.alaudah@microsoft.com> Co-authored-by: Fatemeh <fazamani@microsoft.com> * typos Co-authored-by: yalaudah <yazeed.alaudah@microsoft.com> Co-authored-by: Fatemeh <fazamani@microsoft.com> * tensorboard notebook fix & loading of pre-trained models fix (#397) Co-authored-by: Max Kaznady <max.kaznady@gmail.com> * Docker README corrections and pretrained model checking (#398) * added better instructions to Docker readme; removed HRNet references * added checking of pre-trained models on startup * Update docker/README.md Co-authored-by: yalaudah <yazeed.alaudah@microsoft.com> * added more README changes and a video link with overview * readme tweaks Co-authored-by: yalaudah <yazeed.alaudah@microsoft.com> * finalized performance metrics (#399) Co-authored-by: yalaudah <yazeed.alaudah@microsoft.com> Co-authored-by: Fatemeh <fazamani@microsoft.com> Co-authored-by: Sharat Chikkerur <sharat.chikkerur@microsoft.com> Co-authored-by: kirasoderstrom <kirasoderstrom@gmail.com> Co-authored-by: Sharat Chikkerur <sharat.chikkerur@gmail.com> Co-authored-by: Geisa Faustino <32823639+GeisaFaustino@users.noreply.github.com> Co-authored-by: Ricardo Squassina Lee <8495707+squassina@users.noreply.github.com> Co-authored-by: Michael Zawacki <mikezawacki@hotmail.com> Co-authored-by: Anna Zietlow <annamzietlow@gmail.com> Co-authored-by: Max Kaznady <max.kaznady@gmail.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes:
Includes changes for: