This repository has been archived by the owner on Nov 16, 2023. It is now read-only.
Releases: microsoft/seismic-deeplearning
Releases · microsoft/seismic-deeplearning
0.2.1 Release
Our goal was to finalize performance metrics and update pre-trained models with the latest enhancements from the 0.2 June Release.
Documentation
Bugfixes
0.2 June Release
June 2020 release
Our goal was to add user-facing features in this release, while continuing to harden the correctness of the codebase from the previous release. Developers can now train all models on multiple GPUs and can participate in Bring Your Own Data (BYOD) experience, where any 3D seismic dataset can be loaded with very little effort using a sample script, which demos Penobscot dataset BYOD procedure. This added functionality also means that all new datasets such as Penobscot gain multi-GPU training, which can also be run inside a Docker container. Enhancements such as extra TensorBoard logging metrics also carry over to this release as we have achieved 100% code reusability in the repository across all dataset, model, and training configurations. We drastically reduced the runtime of Pull Request builds by using a synthetic checkerboard testing dataset with stringent unit tests. Quality of training data has also been improved through more thorough sampling of bottom seismic layers. Default model used as SOTA in the repository has been switched from HRNet to SEResNet Unet architecture.
This release also incorporated contributions from the CSE team, which include:
- SEGY to numpy converter utility and sample notebooks, which demonstrate how to work with rectangular boundary SEGY datasets; once converted to numpy, these datasets gain the plug-n-play functionality in the bring-your-own-data (BYOD) experience outlined above (unit tests are included)
- Enhanced ability to perform data ablation studies in the data preprocessing scripts on all datasets - specifically one can specify how many inline and/or crossline slices to remove during training and measure model performance as a function of data sparsity at cross-section level (unit tests are included)
New Features:
- Multi-GPU training both inside and outside of Docker #359 #378 #320
- BYOD with Penobscot example #351 #321
Enhancements:
- Docker image building test #358
- Data loader API enhancement to pass config parameters object to lower-level loaders #352
- Change default model to SEResNet UNet #350
- Merge contributions from CSE team #349
- Enhanced training data quality through more efficient sampling, and depth-wise padding #259
- Documentation enhancements #384 #383 #381 #354
Bugfixes:
Release Notes V00.01.00003
May 2020 release, our goal has been on the robustness and reproducibility of the results provided by the repository. For test coverage, we added synthetic binary and gradient datasets in addition to checkerboard dataset and made specific investments to fix any inconsistencies which were made obvious by running these two extra synthetic datasets through the codebase. We further added more stringent metric tests when training and scoring models on checkerboard synthetic dataset, guaranteeing model execution correctness for patch DeconvNet, UNet and SEResNet UNet models. We also replaced default SOTA HRNet model with SEResNet UNet for all experiments.
Enhancements:
- Binary and gradient synthetic testing dataset #318, #333
- Updated documentation to make the repo more user friendly #315, #332
- Colormap support for any number of classes #324
- Replace default HRNet model with SOTA SEResNet Unet #343
Bugfixes:
Release Notes v0.1.2
April 2020 release, our goal was to harden the current capabilities in the repository and ensure correctness in the results. We made specific investments this month to make sure users of the repository can produce the stated results and reproduce them. Developers can now leverage Docker and Jupyter Lab to train and develop on any platform. We have added a checkerboard dataset which will helps users validate and test the code. Users of the repository can now train and validate data in either the inline or crossline directions, or both simultaneously. We have also improved the Tensorboard visualizations and added the capability to export intermediate data in debug mode.
New features:
- Docker image now ships with Jupyter Lab to allow users to train and develop code on any platform. #237 #255 #242
- Added training and validation set logging to Tensorboard with better visualizations: also refactored the
train.pyscript for Dutchf3 to enhance readability, maintainability. #236 - Added the ability to train and validate the data in the inline or crossline directions only. #253
- Added a new checkerboard dataset to allow users to assess the correctness of the codebase. #249
Enhancements:
- Multi-GPU parallel builds: speeds up build execution for Pull Requests. #246
- Completely removed OpenCV as a dependency from the DeepSeismic Repo. #240
- Removes the
HorovodLRSchedulerclass since its no longer used. #240
Bugfixes:
- Fixed an issue where the generated training and validation splits can overlap (in different directions). #233 #143
- Hardcoded number of classes in Dutch F3 dataloader: can now specify a variable number of classes for Dutch F3 dataset, e.g. map some classes as a single class; also allows for checkerboard dataset or any other dataset to be loaded through Dutch F3 format. #144 #285 #145
- Fixed a visualization bug where the seismic images were not normalized correctly. #217
- Fixed a visualization bug where the predictions were not masked where the input image was padded. This improves the ability to visually inspect and evaluate the results. This solves Issue #230.
- Fixes a potential issue where Tensorboard can crash when a large training batchsize is used. Now the number of images visualized in Tensorboard from every batch has an upper limit.
- Fixes Issue where the epoch number for the images in Tensorboard was always logged as 1 (therefore, not allowing the user to see the epoch number of the different results in Tensorboard. #218
- Fixes an issue where the seeds used for the notebooks are not the same as those used for the training scripts #244
- Fixed an issue where code in
prepare_dutchf3.pywasn't using the stride to create patch locations #287 - Fixed an issue where the borders of images shown in Tensorboard were not displayed correctly #230
v0.1.1
Added Docker support to master. Can now run notebooks and experiments from Jupyter Lab from within Docker.
0.1
final public release PR (#125) * Merged PR 42: Python package structure Created Python package structure * Merged PR 50: Röth-Tarantola generative model for velocities - Created Python package structure for generative models for velocities - Implemented the [Röth-Tarantola model](https://doi.org/10.1029/93JB01563) * Merged PR 51: Isotropic AWE forward modelling using Devito Implemented forward modelling for the isotropic acoustic wave equation using [Devito](https://www.devitoproject.org/) * Merged PR 52: PRNG seed Exposed PRNG seed in generative models for velocities * Merged PR 53: Docs update - Updated LICENSE - Added Microsoft Open Source Code of Conduct - Added Contributing section to README * Merged PR 54: CLI for velocity generators Implemented CLI for velocity generators * Merged PR 69: CLI subpackage using Click Reimplemented CLI as subpackage using Click * Merged PR 70: VS Code settings Added VS Code settings * Merged PR 73: CLI for forward modelling Implemented CLI for forward modelling * Merged PR 76: Unit fixes - Changed to use km/s instead of m/s for velocities - Fixed CLI interface * Merged PR 78: Forward modelling CLI fix * Merged PR 85: Version 0.1.0 * Merging work on salt dataset * Adds computer vision to dependencies * Updates dependecies * Update * Updates the environemnt files * Updates readme and envs * Initial running version of dutchf3 * INFRA: added structure templates. * VOXEL: initial rough code push - need to clean up before PRing. * Working version * Working version before refactor * adding cgmanifest to staging * adding a yml file with CG build task * added prelim NOTICE file * quick minor fixes in README * 3D SEG: first commit for PR. * 3D SEG: removed data files to avoid redistribution. * Merged PR 126: updated notice file with previously excluded components updated notice file with previously excluded components * Updates * 3D SEG: restyled batch file, moving onto others. * Working HRNet * 3D SEG: finished going through Waldeland code * Updates test scripts and makes it take processing arguments * minor update * Fixing imports * Refactoring the experiments * Removing .vscode * Updates gitignore * Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py This PR includes the following changes: - added README instructions for running f3dutch experiments - prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic. - ran black formatter on the file, which created all the formatting changes (sorry!) * Merged PR 204: Adds loaders to deepseismic from cv_lib * Merged PR 209: changes to section loaders in data.py Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts: - get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders. - SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py h, w = img.shape[-2], img.shape[-1] # height and width * Merged PR 210: BENCHMARKS: added placeholder for benchmarks. BENCHMARKS: added placeholder for benchmarks. * Merged PR 211: Fixes issues left over from changes to data.py * Merged PR 220: Adds Horovod and fixes Add Horovod training script Updates dependencies in Horovod docker file Removes hard coding of path in data.py * Merged PR 222: Moves cv_lib into repo and updates setup instructions * Merged PR 236: Cleaned up dutchf3 data loaders @<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments. The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders. This will affect your code if you access these attributes. E.g. if you have something like this in your experiments: ``` train_set = TrainPatchLoader(…) patches = train_set.patches[train_set.split] ``` or ``` train_set = TrainSectionLoader(…) sections = train_set.sections[train_set.split] ``` * Updates the repo with preliminary results for 2D segmentation * Merged PR 248: Experiment: section-based Alaudah training/testing This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment. * Merged PR 253: Waldeland based voxel loaders and TextureNet model Related work items: #16357 * Merged PR 290: A demo notebook on local train/eval on F3 data set Notebook and associated files + minor change in a patch_deconvnet_skip.py model file. Related work items: #17432 * Merged PR 312: moved dutchf3_section to experiments/interpretation moved dutchf3_section to experiments/interpretation Related work items: #17683 * Merged PR 309: minor change to README to reflect the changes in prepare_data script minor change to README to reflect the changes in prepare_data script Related work items: #17681 * Merged PR 315: Removing voxel exp Related work items: #17702 * Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo. Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script. Related work items: #18264 * Merged PR 405: minor mods to notebook, more documentation A very small PR - Just a few more lines of documentation in the notebook, to improve clarity. Related work items: #17432 * Merged PR 368: Adds penobscot Adds for penobscot - Dataset reader - Training script - Testing script - Section depth augmentation - Patch depth augmentation - Iinline visualisation for Tensorboard Related work items: #14560, #17697, #17699, #17700 * Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators Related work items: #16362 * Merged PR 452: decouple docker image creation from azureml removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb All other changes are due to trivial reruns Related work items: #18346 * Merged PR 512: Pre-commit hooks for formatting and style checking Opening this PR to start the discussion - I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added: - .pre-commit-config.yaml - defines git hooks to be installed - .flake8 - settings for flake8 linter - pyproject.toml - settings for black formatter The last two files define the formatting and linting style we want to enforce on the repo. All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors. Some questions to start the discussion: - Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that. - Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file. - Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this? Thanks! Related work items: #18350 * Merged PR 513: 3D training script for Waldeland's model with Ignite Related work items: #16356 * Merged PR 565: Demo notebook updated with 3D graph Changes: 1) Updated demo notebook with the 3D visualization 2) Formatting changes due to new black/flake8 git hook Related work items: #17432 * Merged PR 569: Minor PR: change to pre-commit configuration files Related work items: #18350 * Merged PR 586: Purging unused files and experiments Purging unused files and experiments Related work items: #20499 * Merged PR 601: Fixes to penobscot experiments A few changes: - Instructions in README on how to download and process Penobscot and F3 2D data sets - moved prepare_data scripts to the scripts/ directory - fixed a weird issue with a class method in Penobscot data loader - fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue) - removed config files that were not tested or working in Penobscot experiments - modified default.py so it's working if train.py ran without a config file Related work items: #20694 * Merged PR 605: added common metrics to Waldeland model in Ignite Related work items: #19550 * added cela copyright headers to all non-empty .py files (#3) * switched to ACR instead of docker hub (#4) * sdk.v1.0.69, plus switched to ACR push. ACR pull coming next * full acr use, push and pull, and use in Estimator * temp fix for dcker image bug * fixed the az acr login --username and --password issue * full switch to ACR for docker image storage * Vapaunic/metrics (#1) * Merging work on salt dataset * Adds computer vision to dependencies * Updates dependecies * Update * Updates the environemnt files * Updates readme and envs * Initial running version of dutchf3 * INFRA: added structure templates. * VOXEL: initial rough code push - need to clean up before PRing. * Working version * Working version before refactor * quick minor fixes in README * 3D SEG: first commit for PR. * 3D SEG: removed data files to avoid redistribution. * Updates * 3D SEG: restyled batch file, moving onto others. * Working HRNet * 3D SEG: finished going through Waldeland code * Updates test scripts and makes it take processing arguments * minor update * Fixing imports * Refactoring the experiments * Removing .vscode * Updates gitignore * added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script * added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script * minor wording fix * minor wording fix * enabled splitting dataset into sections, rather than only patches * enabled splitting dataset into sections, rather than only patches * merged duplicate ifelse blocks * merged duplicate ifelse blocks * refactored prepare_data.py * refactored prepare_data.py * added scripts for section train test * added scripts for section train test * section train/test works for single channel input * section train/test works for single channel input * Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py This PR includes the following changes: - added README instructions for running f3dutch experiments - prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic. - ran black formatter on the file, which created all the formatting changes (sorry!) * Merged PR 204: Adds loaders to deepseismic from cv_lib * train and test script for section based training/testing * train and test script for section based training/testing * Merged PR 209: changes to section loaders in data.py Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts: - get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders. - SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py h, w = img.shape[-2], img.shape[-1] # height and width * Merged PR 210: BENCHMARKS: added placeholder for benchmarks. BENCHMARKS: added placeholder for benchmarks. * Merged PR 211: Fixes issues left over from changes to data.py * removing experiments from deep_seismic, following the new struct * removing experiments from deep_seismic, following the new struct * Merged PR 220: Adds Horovod and fixes Add Horovod training script Updates dependencies in Horovod docker file Removes hard coding of path in data.py * section train/test scripts * section train/test scripts * Add cv_lib to repo and updates instructions * Add cv_lib to repo and updates instructions * Removes data.py and updates readme * Removes data.py and updates readme * Updates requirements * Updates requirements * Merged PR 222: Moves cv_lib into repo and updates setup instructions * renamed train/test scripts * renamed train/test scripts * train test works on alaudah section experiments, a few minor bugs left * train test works on alaudah section experiments, a few minor bugs left * cleaning up loaders * cleaning up loaders * Merged PR 236: Cleaned up dutchf3 data loaders @<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments. The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders. This will affect your code if you access these attributes. E.g. if you have something like this in your experiments: ``` train_set = TrainPatchLoader(…) patches = train_set.patches[train_set.split] ``` or ``` train_set = TrainSectionLoader(…) sections = train_set.sections[train_set.split] ``` * training testing for sections works * training testing for sections works * minor changes * minor changes * reverting changes on dutchf3/local/default.py file * reverting changes on dutchf3/local/default.py file * added config file * added config file * Updates the repo with preliminary results for 2D segmentation * Merged PR 248: Experiment: section-based Alaudah training/testing This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment. * Merged PR 253: Waldeland based voxel loaders and TextureNet model Related work items: #16357 * Merged PR 290: A demo notebook on local train/eval on F3 data set Notebook and associated files + minor change in a patch_deconvnet_skip.py model file. Related work items: #17432 * Merged PR 312: moved dutchf3_section to experiments/interpretation moved dutchf3_section to experiments/interpretation Related work items: #17683 * Merged PR 309: minor change to README to reflect the changes in prepare_data script minor change to README to reflect the changes in prepare_data script Related work items: #17681 * Merged PR 315: Removing voxel exp Related work items: #17702 * sync with new experiment structure * sync with new experiment structure * added a logging handler for array metrics * added a logging handler for array metrics * first draft of metrics based on the ignite confusion matrix * first draft of metrics based on the ignite confusion matrix * metrics now based on ignite.metrics * metrics now based on ignite.metrics * modified patch train.py with new metrics * modified patch train.py with new metrics * Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo. Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script. Related work items: #18264 * modified metrics with ignore_index * modified metrics with ignore_index * Merged PR 405: minor mods to notebook, more documentation A very small PR - Just a few more lines of documentation in the notebook, to improve clarity. Related work items: #17432 * Merged PR 368: Adds penobscot Adds for penobscot - Dataset reader - Training script - Testing script - Section depth augmentation - Patch depth augmentation - Iinline visualisation for Tensorboard Related work items: #14560, #17697, #17699, #17700 * Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators Related work items: #16362 * Merged PR 452: decouple docker image creation from azureml removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb All other changes are due to trivial reruns Related work items: #18346 * Merged PR 512: Pre-commit hooks for formatting and style checking Opening this PR to start the discussion - I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added: - .pre-commit-config.yaml - defines git hooks to be installed - .flake8 - settings for flake8 linter - pyproject.toml - settings for black formatter The last two files define the formatting and linting style we want to enforce on the repo. All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors. Some questions to start the discussion: - Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that. - Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file. - Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this? Thanks! Related work items: #18350 * Merged PR 513: 3D training script for Waldeland's model with Ignite Related work items: #16356 * Merged PR 565: Demo notebook updated with 3D graph Changes: 1) Updated demo notebook with the 3D visualization 2) Formatting changes due to new black/flake8 git hook Related work items: #17432 * Merged PR 341: Tests for cv_lib/metrics This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged. I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing. Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest. Related work items: #16955 * Merged PR 341: Tests for cv_lib/metrics This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged. I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing. Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest. Related work items: #16955 * merged tests into this branch * merged tests into this branch * Merged PR 569: Minor PR: change to pre-commit configuration files Related work items: #18350 * Merged PR 586: Purging unused files and experiments Purging unused files and experiments Related work items: #20499 * moved prepare data under scripts * moved prepare data under scripts * removed untested model configs * removed untested model configs * fixed weird bug in penobscot data loader * fixed weird bug in penobscot data loader * penobscot experiments working for hrnet, seresnet, no depth and patch depth * penobscot experiments working for hrnet, seresnet, no depth and patch depth * removed a section loader bug in the penobscot loader * removed a section loader bug in the penobscot loader * removed a section loader bug in the penobscot loader * removed a section loader bug in the penobscot loader * fixed bugs in my previous 'fix' * fixed bugs in my previous 'fix' * removed redundant _open_mask from subclasses * removed redundant _open_mask from subclasses * Merged PR 601: Fixes to penobscot experiments A few changes: - Instructions in README on how to download and process Penobscot and F3 2D data sets - moved prepare_data scripts to the scripts/ directory - fixed a weird issue with a class method in Penobscot data loader - fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue) - removed config files that were not tested or working in Penobscot experiments - modified default.py so it's working if train.py ran without a config file Related work items: #20694 * Merged PR 605: added common metrics to Waldeland model in Ignite Related work items: #19550 * Removed redundant extract_metric_from * Removed redundant extract_metric_from * formatting changes in metrics * formatting changes in metrics * modified penobscot experiment to use new local metrics * modified penobscot experiment to use new local metrics * modified section experimen to pass device to metrics * modified section experimen to pass device to metrics * moved metrics out of dutchf3, modified distributed to work with the new metrics * moved metrics out of dutchf3, modified distributed to work with the new metrics * fixed other experiments after new metrics * fixed other experiments after new metrics * removed apex metrics from distributed train.py * removed apex metrics from distributed train.py * added ignite-based metrics to dutch voxel experiment * added ignite-based metrics to dutch voxel experiment * removed apex metrics * removed apex metrics * modified penobscot test script to use new metrics * pytorch-ignite pre-release with new metrics until stable available * removed cell output from the F3 notebook * deleted .vscode * modified metric import in test_metrics.py * separated metrics out as a module * BUILD: added build setup files. (#5) * Update main_build.yml for Azure Pipelines * BUILD: added build status badges (#6) * Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7) * Finished version of numpy data loader * Working training script for demo * Adds the new metrics * Fixes docstrings and adds header * Removing extra setup.py * Log config file now experiment specific (#8) * Merging work on salt dataset * Adds computer vision to dependencies * Updates dependecies * Update * Updates the environemnt files * Updates readme and envs * Initial running version of dutchf3 * INFRA: added structure templates. * VOXEL: initial rough code push - need to clean up before PRing. * Working version * Working version before refactor * quick minor fixes in README * 3D SEG: first commit for PR. * 3D SEG: removed data files to avoid redistribution. * Updates * 3D SEG: restyled batch file, moving onto others. * Working HRNet * 3D SEG: finished going through Waldeland code * Updates test scripts and makes it take processing arguments * minor update * Fixing imports * Refactoring the experiments * Removing .vscode * Updates gitignore * added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script * added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script * minor wording fix * minor wording fix * enabled splitting dataset into sections, rather than only patches * enabled splitting dataset into sections, rather than only patches * merged duplicate ifelse blocks * merged duplicate ifelse blocks * refactored prepare_data.py * refactored prepare_data.py * added scripts for section train test * added scripts for section train test * section train/test works for single channel input * section train/test works for single channel input * Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py This PR includes the following changes: - added README instructions for running f3dutch experiments - prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic. - ran black formatter on the file, which created all the formatting changes (sorry!) * Merged PR 204: Adds loaders to deepseismic from cv_lib * train and test script for section based training/testing * train and test script for section based training/testing * Merged PR 209: changes to section loaders in data.py Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts: - get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders. - SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py h, w = img.shape[-2], img.shape[-1] # height and width * Merged PR 210: BENCHMARKS: added placeholder for benchmarks. BENCHMARKS: added placeholder for benchmarks. * Merged PR 211: Fixes issues left over from changes to data.py * removing experiments from deep_seismic, following the new struct * removing experiments from deep_seismic, following the new struct * Merged PR 220: Adds Horovod and fixes Add Horovod training script Updates dependencies in Horovod docker file Removes hard coding of path in data.py * section train/test scripts * section train/test scripts * Add cv_lib to repo and updates instructions * Add cv_lib to repo and updates instructions * Removes data.py and updates readme * Removes data.py and updates readme * Updates requirements * Updates requirements * Merged PR 222: Moves cv_lib into repo and updates setup instructions * renamed train/test scripts * renamed train/test scripts * train test works on alaudah section experiments, a few minor bugs left * train test works on alaudah section experiments, a few minor bugs left * cleaning up loaders * cleaning up loaders * Merged PR 236: Cleaned up dutchf3 data loaders @<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments. The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders. This will affect your code if you access these attributes. E.g. if you have something like this in your experiments: ``` train_set = TrainPatchLoader(…) patches = train_set.patches[train_set.split] ``` or ``` train_set = TrainSectionLoader(…) sections = train_set.sections[train_set.split] ``` * training testing for sections works * training testing for sections works * minor changes * minor changes * reverting changes on dutchf3/local/default.py file * reverting changes on dutchf3/local/default.py file * added config file * added config file * Updates the repo with preliminary results for 2D segmentation * Merged PR 248: Experiment: section-based Alaudah training/testing This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment. * Merged PR 253: Waldeland based voxel loaders and TextureNet model Related work items: #16357 * Merged PR 290: A demo notebook on local train/eval on F3 data set Notebook and associated files + minor change in a patch_deconvnet_skip.py model file. Related work items: #17432 * Merged PR 312: moved dutchf3_section to experiments/interpretation moved dutchf3_section to experiments/interpretation Related work items: #17683 * Merged PR 309: minor change to README to reflect the changes in prepare_data script minor change to README to reflect the changes in prepare_data script Related work items: #17681 * Merged PR 315: Removing voxel exp Related work items: #17702 * sync with new experiment structure * sync with new experiment structure * added a logging handler for array metrics * added a logging handler for array metrics * first draft of metrics based on the ignite confusion matrix * first draft of metrics based on the ignite confusion matrix * metrics now based on ignite.metrics * metrics now based on ignite.metrics * modified patch train.py with new metrics * modified patch train.py with new metrics * Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo. Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script. Related work items: #18264 * modified metrics with ignore_index * modified metrics with ignore_index * Merged PR 405: minor mods to notebook, more documentation A very small PR - Just a few more lines of documentation in the notebook, to improve clarity. Related work items: #17432 * Merged PR 368: Adds penobscot Adds for penobscot - Dataset reader - Training script - Testing script - Section depth augmentation - Patch depth augmentation - Iinline visualisation for Tensorboard Related work items: #14560, #17697, #17699, #17700 * Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators Related work items: #16362 * Merged PR 452: decouple docker image creation from azureml removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb All other changes are due to trivial reruns Related work items: #18346 * Merged PR 512: Pre-commit hooks for formatting and style checking Opening this PR to start the discussion - I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added: - .pre-commit-config.yaml - defines git hooks to be installed - .flake8 - settings for flake8 linter - pyproject.toml - settings for black formatter The last two files define the formatting and linting style we want to enforce on the repo. All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors. Some questions to start the discussion: - Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that. - Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file. - Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this? Thanks! Related work items: #18350 * Merged PR 513: 3D training script for Waldeland's model with Ignite Related work items: #16356 * Merged PR 565: Demo notebook updated with 3D graph Changes: 1) Updated demo notebook with the 3D visualization 2) Formatting changes due to new black/flake8 git hook Related work items: #17432 * Merged PR 341: Tests for cv_lib/metrics This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged. I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing. Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest. Related work items: #16955 * Merged PR 341: Tests for cv_lib/metrics This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged. I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing. Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest. Related work items: #16955 * merged tests into this branch * merged tests into this branch * Merged PR 569: Minor PR: change to pre-commit configuration files Related work items: #18350 * Merged PR 586: Purging unused files and experiments Purging unused files and experiments Related work items: #20499 * moved prepare data under scripts * moved prepare data under scripts * removed untested model configs * removed untested model configs * fixed weird bug in penobscot data loader * fixed weird bug in penobscot data loader * penobscot experiments working for hrnet, seresnet, no depth and patch depth * penobscot experiments working for hrnet, seresnet, no depth and patch depth * removed a section loader bug in the penobscot loader * removed a section loader bug in the penobscot loader * removed a section loader bug in the penobscot loader * removed a section loader bug in the penobscot loader * fixed bugs in my previous 'fix' * fixed bugs in my previous 'fix' * removed redundant _open_mask from subclasses * removed redundant _open_mask from subclasses * Merged PR 601: Fixes to penobscot experiments A few changes: - Instructions in README on how to download and process Penobscot and F3 2D data sets - moved prepare_data scripts to the scripts/ directory - fixed a weird issue with a class method in Penobscot data loader - fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue) - removed config files that were not tested or working in Penobscot experiments - modified default.py so it's working if train.py ran without a config file Related work items: #20694 * Merged PR 605: added common metrics to Waldeland model in Ignite Related work items: #19550 * Removed redundant extract_metric_from * Removed redundant extract_metric_from * formatting changes in metrics * formatting changes in metrics * modified penobscot experiment to use new local metrics * modified penobscot experiment to use new local metrics * modified section experimen to pass device to metrics * modified section experimen to pass device to metrics * moved metrics out of dutchf3, modified distributed to work with the new metrics * moved metrics out of dutchf3, modified distributed to work with the new metrics * fixed other experiments after new metrics * fixed other experiments after new metrics * removed apex metrics from distributed train.py * removed apex metrics from distributed train.py * added ignite-based metrics to dutch voxel experiment * added ignite-based metrics to dutch voxel experiment * removed apex metrics * removed apex metrics * modified penobscot test script to use new metrics * pytorch-ignite pre-release with new metrics until stable available * removed cell output from the F3 notebook * deleted .vscode * modified metric import in test_metrics.py * separated metrics out as a module * relative logger file path, modified section experiment * removed the REPO_PATH from init * created util logging function, and moved logging file to each experiment * modified demo experiment * modified penobscot experiment * modified dutchf3_voxel experiment * no logging in voxel2pixel * modified dutchf3 patch local experiment * modified patch distributed experiment * modified interpretation notebook * minor changes to comments * DOC: forking dislaimer and new build names. (#9) * Updating README.md with introduction material (#10) * Update README with introduction to DeepSeismic Add intro material for DeepSeismic * Adding logo file * Adding image to readme * Update README.md * Updates the 3D visualisation to use itkwidgets (#11) * Updates notebook to use itkwidgets for interactive visualisation * Adds jupytext to pre-commit (#12) * Add jupytext * Adds demo notebook for HRNet (#13) * Adding TF 2.0 to allow for tensorboard vis in notebooks * Modifies hrnet config for notebook * Add HRNet notebook for demo * Updates HRNet notebook and tidies F3 * removed my username references (#15) * moving 3D models into contrib folder (#16) * Weetok (#17) * Update it to include sections for imaging * Update README.md * Update README.md * added pytest to environmetn, and pytest job to the main build (#18) * Update main_build.yml for Azure Pipelines * minor stylistic changes (#19) * Update main_build.yml for Azure Pipelines Added template for integration tests for scripts and experiments Added setup and env Increased job timeout added complete set of tests * BUILD: placeholder for Azure pipelines for notebooks build. BUILD: added notebooks job placeholders. BUILD: added github badgets for notebook builds * CLEANUP: moved non-release items to contrib (#20) * Updates HRNet notebook 🚀 (#25) * Modifies pre-commit hook to modify output * Modifies the HRNet notebook to use Penobscot dataset Adds parameters to limit iterations Adds parameters meta tag for papermil * Fixing merge peculiarities * Updates environment.yaml (#21) * Pins main libraries Adds cudatoolkit version based on issues faced during workshop * removing files * Updates Readme (#22) * Adds model instructions to readme * Update README.md (#24) I have collected points to all of our BP repos into this central place. We are trying to create links between everything to draw people from one to the other. Can we please add a pointer here to the readme? I have spoken with Max and will be adding Deep Seismic there once you have gone public. * CONTRIB: cleanup for imaging. (#28) * Create Unit Test Build.yml (#29) Adding Unit Test Build. * Update README.md * Update README.md * Create Unit Test Build.yml (#29) Adding Unit Test Build. Update README.md Update README.md * azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing * azureml sdk 1.0.74; foxed a few issues around ACR access; added nb 030 for scalability testing * TESTS: added notebook integreation tests. (#65) * TESTS: added notebook integreation tests. * TEST: typo in env name * Addressing a number of minor issues with README and broken links (#67) * Update main_build.yml for Azure Pipelines * Update main_build.yml for Azure Pipelines * BUILD: added build status badges (#6) * Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7) * Finished version of numpy data loader * Working training script for demo * Adds the new metrics * Fixes docstrings and adds header * Removing extra setup.py * Log config file now experiment specific (#8) * Merging work on salt dataset * Adds computer vision to dependencies * Updates dependecies * Update * Updates the environemnt files * Updates readme and envs * Initial running version of dutchf3 * INFRA: added structure templates. * VOXEL: initial rough code push - need to clean up before PRing. * Working version * Working version before refactor * quick minor fixes in README * 3D SEG: first commit for PR. * 3D SEG: removed data files to avoid redistribution. * Updates * 3D SEG: restyled batch file, moving onto others. * Working HRNet * 3D SEG: finished going through Waldeland code * Updates test scripts and makes it take processing arguments * minor update * Fixing imports * Refactoring the experiments * Removing .vscode * Updates gitignore * added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script * added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script * minor wording fix * minor wording fix * enabled splitting dataset into sections, rather than only patches * enabled splitting dataset into sections, rather than only patches * merged duplicate ifelse blocks * merged duplicate ifelse blocks * refactored prepare_data.py * refactored prepare_data.py * added scripts for section train test * added scripts for section train test * section train/test works for single channel input * section train/test works for single channel input * Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py This PR includes the following changes: - added README instructions for running f3dutch experiments - prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic. - ran black formatter on the file, which created all the formatting changes (sorry!) * Merged PR 204: Adds loaders to deepseismic from cv_lib * train and test script for section based training/testing * train and test script for section based training/testing * Merged PR 209: changes to section loaders in data.py Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts: - get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders. - SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py h, w = img.shape[-2], img.shape[-1] # height and width * Merged PR 210: BENCHMARKS: added placeholder for benchmarks. BENCHMARKS: added placeholder for benchmarks. * Merged PR 211: Fixes issues left over from changes to data.py * removing experiments from deep_seismic, following the new struct * removing experiments from deep_seismic, following the new struct * Merged PR 220: Adds Horovod and fixes Add Horovod training script Updates dependencies in Horovod docker file Removes hard coding of path in data.py * section train/test scripts * section train/test scripts * Add cv_lib to repo and updates instructions * Add cv_lib to repo and updates instructions * Removes data.py and updates readme * Removes data.py and updates readme * Updates requirements * Updates requirements * Merged PR 222: Moves cv_lib into repo and updates setup instructions * renamed train/test scripts * renamed train/test scripts * train test works on alaudah section experiments, a few minor bugs left * train test works on alaudah section experiments, a few minor bugs left * cleaning up loaders * cleaning up loaders * Merged PR 236: Cleaned up dutchf3 data loaders @<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments. The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders. This will affect your code if you access these attributes. E.g. if you have something like this in your experiments: ``` train_set = TrainPatchLoader(…) patches = train_set.patches[train_set.split] ``` or ``` train_set = TrainSectionLoader(…) sections = train_set.sections[train_set.split] ``` * training testing for sections works * training testing for sections works * minor changes * minor changes * reverting changes on dutchf3/local/default.py file * reverting changes on dutchf3/local/default.py file * added config file * added config file * Updates the repo with preliminary results for 2D segmentation * Merged PR 248: Experiment: section-based Alaudah training/testing This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment. * Merged PR 253: Waldeland based voxel loaders and TextureNet model Related work items: #16357 * Merged PR 290: A demo notebook on local train/eval on F3 data set Notebook and associated files + minor change in a patch_deconvnet_skip.py model file. Related work items: #17432 * Merged PR 312: moved dutchf3_section to experiments/interpretation moved dutchf3_section to experiments/interpretation Related work items: #17683 * Merged PR 309: minor change to README to reflect the changes in prepare_data script minor change to README to reflect the changes in prepare_data script Related work items: #17681 * Merged PR 315: Removing voxel exp Related work items: #17702 * sync with new experiment structure * sync with new experiment structure * added a logging handler for array metrics * added a logging handler for array metrics * first draft of metrics based on the ignite confusion matrix * first draft of metrics based on the ignite confusion matrix * metrics now based on ignite.metrics * metrics now based on ignite.metrics * modified patch train.py with new metrics * modified patch train.py with new metrics * Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo. Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script. Related work items: #18264 * modified metrics with ignore_index * modified metrics with ignore_index * Merged PR 405: minor mods to notebook, more documentation A very small PR - Just a few more lines of documentation in the notebook, to improve clarity. Related work items: #17432 * Merged PR 368: Adds penobscot Adds for penobscot - Dataset reader - Training script - Testing script - Section depth augmentation - Patch depth augmentation - Iinline visualisation for Tensorboard Related work items: #14560, #17697, #17699, #17700 * Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators Related work items: #16362 * Merged PR 452: decouple docker image creation from azureml removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb All other changes are due to trivial reruns Related work items: #18346 * Merged PR 512: Pre-commit hooks for formatting and style checking Opening this PR to start the discussion - I added the required dotenv files and instructions for setting up pre-commit hooks for formatting and style checking. For formatting, we are using black, and style checking flake8. The following files are added: - .pre-commit-config.yaml - defines git hooks to be installed - .flake8 - settings for flake8 linter - pyproject.toml - settings for black formatter The last two files define the formatting and linting style we want to enforce on the repo. All of us would set up the pre-commit hooks locally, so regardless of what formatting/linting settings we have in our local editors, the settings specified by the git hooks would still be enforced prior to the commit, to ensure consistency among contributors. Some questions to start the discussion: - Do you want to change any of the default settings in the dotenv files - like the line lengths, error messages we exclude or include, or anything like that. - Do we want to have a requirements-dev.txt file for contributors? This setup uses pre-commit package, I didn't include it in the environment.yaml file, but instead instructed the user to install it in the CONTRIBUTING.MD file. - Once you have the hooks installed, it will only affect the files you are committing in the future. A big chunk of our codebase does not conform to the formatting/style settings. We will have to run the hooks on the codebase retrospectively. I'm happy to do that, but it will create many changes and a significant looking PR :) Any thoughts on how we should approach this? Thanks! Related work items: #18350 * Merged PR 513: 3D training script for Waldeland's model with Ignite Related work items: #16356 * Merged PR 565: Demo notebook updated with 3D graph Changes: 1) Updated demo notebook with the 3D visualization 2) Formatting changes due to new black/flake8 git hook Related work items: #17432 * Merged PR 341: Tests for cv_lib/metrics This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged. I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing. Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest. Related work items: #16955 * Merged PR 341: Tests for cv_lib/metrics This PR is dependent on the tests created in the previous branch !333. That's why the PR is to merge tests into vapaunic/metrics branch (so the changed files below only include the diff between these two branches. However, I can change this once the vapaunic/metrics is merged. I created these tests under cv_lib/ since metrics are a part of that library. I imagine we will have tests under deepseismic_interpretation/, and the top level /tests for integration testing. Let me know if you have any comments on this test, or the structure. As agreed, I'm using pytest. Related work items: #16955 * merged tests into this branch * merged tests into this branch * Merged PR 569: Minor PR: change to pre-commit configuration files Related work items: #18350 * Merged PR 586: Purging unused files and experiments Purging unused files and experiments Related work items: #20499 * moved prepare data under scripts * moved prepare data under scripts * removed untested model configs * removed untested model configs * fixed weird bug in penobscot data loader * fixed weird bug in penobscot data loader * penobscot experiments working for hrnet, seresnet, no depth and patch depth * penobscot experiments working for hrnet, seresnet, no depth and patch depth * removed a section loader bug in the penobscot loader * removed a section loader bug in the penobscot loader * removed a section loader bug in the penobscot loader * removed a section loader bug in the penobscot loader * fixed bugs in my previous 'fix' * fixed bugs in my previous 'fix' * removed redundant _open_mask from subclasses * removed redundant _open_mask from subclasses * Merged PR 601: Fixes to penobscot experiments A few changes: - Instructions in README on how to download and process Penobscot and F3 2D data sets - moved prepare_data scripts to the scripts/ directory - fixed a weird issue with a class method in Penobscot data loader - fixed a bug in section loader (_add_extra_channel in section loader was not necessary and was causing an issue) - removed config files that were not tested or working in Penobscot experiments - modified default.py so it's working if train.py ran without a config file Related work items: #20694 * Merged PR 605: added common metrics to Waldeland model in Ignite Related work items: #19550 * Removed redundant extract_metric_from * Removed redundant extract_metric_from * formatting changes in metrics * formatting changes in metrics * modified penobscot experiment to use new local metrics * modified penobscot experiment to use new local metrics * modified section experimen to pass device to metrics * modified section experimen to pass device to metrics * moved metrics out of dutchf3, modified distributed to work with the new metrics * moved metrics out of dutchf3, modified distributed to work with the new metrics * fixed other experiments after new metrics * fixed other experiments after new metrics * removed apex metrics from distributed train.py * removed apex metrics from distributed train.py * added ignite-based metrics to dutch voxel experiment * added ignite-based metrics to dutch voxel experiment * removed apex metrics * removed apex metrics * modified penobscot test script to use new metrics * pytorch-ignite pre-release with new metrics until stable available * removed cell output from the F3 notebook * deleted .vscode * modified metric import in test_metrics.py * separated metrics out as a module * relative logger file path, modified section experiment * removed the REPO_PATH from init * created util logging function, and moved logging file to each experiment * modified demo experiment * modified penobscot experiment * modified dutchf3_voxel experiment * no logging in voxel2pixel * modified dutchf3 patch local experiment * modified patch distributed experiment * modified interpretation notebook * minor changes to comments * DOC: forking dislaimer and new build names. (#9) * Updating README.md with introduction material (#10) * Update README with introduction to DeepSeismic Add intro material for DeepSeismic * Adding logo file * Adding image to readme * Update README.md * Updates the 3D visualisation to use itkwidgets (#11) * Updates notebook to use itkwidgets for interactive visualisation * Adds jupytext to pre-commit (#12) * Add jupytext * Adds demo notebook for HRNet (#13) * Adding TF 2.0 to allow for tensorboard vis in notebooks * Modifies hrnet config for notebook * Add HRNet notebook for demo * Updates HRNet notebook and tidies F3 * removed my username references (#15) * moving 3D models into contrib folder (#16) * Weetok (#17) * Update it to include sections for imaging * Update README.md * Update README.md * fix for seyviewer and mkdir splits in README + broken link in F3 notebook * issue edits to README * download complete message * Added Yacs info to README.md (#69) * Update main_build.yml for Azure Pipelines * Update main_build.yml for Azure Pipelines * BUILD: added build status badges (#6) * Adds dataloader for numpy datasets as well as demo pipeline for such a dataset (#7) * Finished version of numpy data loader * Working training script for demo * Adds the new metrics * Fixes docstrings and adds header * Removing extra setup.py * Log config file now experiment specific (#8) * Merging work on salt dataset * Adds computer vision to dependencies * Updates dependecies * Update * Updates the environemnt files * Updates readme and envs * Initial running version of dutchf3 * INFRA: added structure templates. * VOXEL: initial rough code push - need to clean up before PRing. * Working version * Working version before refactor * quick minor fixes in README * 3D SEG: first commit for PR. * 3D SEG: removed data files to avoid redistribution. * Updates * 3D SEG: restyled batch file, moving onto others. * Working HRNet * 3D SEG: finished going through Waldeland code * Updates test scripts and makes it take processing arguments * minor update * Fixing imports * Refactoring the experiments * Removing .vscode * Updates gitignore * added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script * added instructions for running f3dutch experiments, and fixed some issues in prepare_data.py script * minor wording fix * minor wording fix * enabled splitting dataset into sections, rather than only patches * enabled splitting dataset into sections, rather than only patches * merged duplicate ifelse blocks * merged duplicate ifelse blocks * refactored prepare_data.py * refactored prepare_data.py * added scripts for section train test * added scripts for section train test * section train/test works for single channel input * section train/test works for single channel input * Merged PR 174: F3 Dutch README, and fixed issues in prepare_data.py This PR includes the following changes: - added README instructions for running f3dutch experiments - prepare_dataset.py didn't work for creating section-based splits, so I fixed a few issues. There are no changes to the patch-based splitting logic. - ran black formatter on the file, which created all the formatting changes (sorry!) * Merged PR 204: Adds loaders to deepseismic from cv_lib * train and test script for section based training/testing * train and test script for section based training/testing * Merged PR 209: changes to section loaders in data.py Changes in this PR will affect patch scripts as well. The following are required changes in patch scripts: - get_train_loader() in train.py should be changed to get_patch_loader(). I created separate function to load section and patch loaders. - SectionLoader now swaps H and W dims. When loading test data in patch, this line can be removed (and tested) from test.py h, w = img.shape[-2], img.shape[-1] # height and width * Merged PR 210: BENCHMARKS: added placeholder for benchmarks. BENCHMARKS: added placeholder for benchmarks. * Merged PR 211: Fixes issues left over from changes to data.py * removing experiments from deep_seismic, following the new struct * removing experiments from deep_seismic, following the new struct * Merged PR 220: Adds Horovod and fixes Add Horovod training script Updates dependencies in Horovod docker file Removes hard coding of path in data.py * section train/test scripts * section train/test scripts * Add cv_lib to repo and updates instructions * Add cv_lib to repo and updates instructions * Removes data.py and updates readme * Removes data.py and updates readme * Updates requirements * Updates requirements * Merged PR 222: Moves cv_lib into repo and updates setup instructions * renamed train/test scripts * renamed train/test scripts * train test works on alaudah section experiments, a few minor bugs left * train test works on alaudah section experiments, a few minor bugs left * cleaning up loaders * cleaning up loaders * Merged PR 236: Cleaned up dutchf3 data loaders @<Mathew Salvaris> , @<Ilia Karmanov> , @<Max Kaznady> , please check out if this PR will affect your experiments. The main change is with the initialization of sections/patches attributes of loaders. Previously, we were unnecessarily assigning all train/val splits to train loaders, rather than only those belonging to the given split for that loader. Similar for test loaders. This will affect your code if you access these attributes. E.g. if you have something like this in your experiments: ``` train_set = TrainPatchLoader(…) patches = train_set.patches[train_set.split] ``` or ``` train_set = TrainSectionLoader(…) sections = train_set.sections[train_set.split] ``` * training testing for sections works * training testing for sections works * minor changes * minor changes * reverting changes on dutchf3/local/default.py file * reverting changes on dutchf3/local/default.py file * added config file * added config file * Updates the repo with preliminary results for 2D segmentation * Merged PR 248: Experiment: section-based Alaudah training/testing This PR includes the section-based experiments on dutchf3 to replicate Alaudah's work. No changes were introduced to the code outside this experiment. * Merged PR 253: Waldeland based voxel loaders and TextureNet model Related work items: #16357 * Merged PR 290: A demo notebook on local train/eval on F3 data set Notebook and associated files + minor change in a patch_deconvnet_skip.py model file. Related work items: #17432 * Merged PR 312: moved dutchf3_section to experiments/interpretation moved dutchf3_section to experiments/interpretation Related work items: #17683 * Merged PR 309: minor change to README to reflect the changes in prepare_data script minor change to README to reflect the changes in prepare_data script Related work items: #17681 * Merged PR 315: Removing voxel exp Related work items: #17702 * sync with new experiment structure * sync with new experiment structure * added a logging handler for array metrics * added a logging handler for array metrics * first draft of metrics based on the ignite confusion matrix * first draft of metrics based on the ignite confusion matrix * metrics now based on ignite.metrics * metrics now based on ignite.metrics * modified patch train.py with new metrics * modified patch train.py with new metrics * Merged PR 361: VOXEL: fixes to original voxel2pixel code to make it work with the rest of the repo. Realized there was one bug in the code and the rest of the functions did not work with the different versions of libraries which we have listed in the conda yaml file. Also updated the download script. Related work items: #18264 * modified metrics with ignore_index * modified metrics with ignore_index * Merged PR 405: minor mods to notebook, more documentation A very small PR - Just a few more lines of documentation in the notebook, to improve clarity. Related work items: #17432 * Merged PR 368: Adds penobscot Adds for penobscot - Dataset reader - Training script - Testing script - Section depth augmentation - Patch depth augmentation - Iinline visualisation for Tensorboard Related work items: #14560, #17697, #17699, #17700 * Merged PR 407: Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators Azure ML SDK Version: 1.0.65; running devito in AzureML Estimators Related work items: #16362 * Merged PR 452: decouple docker image creation from azureml removed all azureml dependencies from 010_CreateExperimentationDockerImage_GeophysicsTutorial_FWI_Azure_devito.ipynb All other changes are due to trivial reruns Related work items: #18346 * Merged PR 512: Pre-commit hooks for formatting and style checking Opening this PR to start the discussion - …