From 34502b709e5a07bb1050ef5618805393e0d35895 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Wed, 27 May 2020 16:51:39 +0000 Subject: [PATCH 1/9] README updates, ran through codebase on vm --- README.md | 8 ++++---- examples/interpretation/README.md | 4 +++- experiments/interpretation/dutchf3_patch/README.md | 4 ++-- 3 files changed, 9 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index eb89c926..d10b200d 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ If you run into any problems, chances are your problem has already been solved i The notebook is designed to be run in demo mode by default using a pre-trained model in under 5 minutes on any reasonable Deep Learning GPU such as nVidia K80/P40/P100/V100/TitanV. ### Azure Machine Learning -[Azure Machine Learning](https://docs.microsoft.com/en-us/azure/machine-learning/) enables you to train and deploy your machine learning models and pipelines at scale, ane leverage open-source Python frameworks, such as PyTorch, TensorFlow, and scikit-learn. If you are looking at getting started with using the code in this repository with Azure Machine Learning, refer to [Azure Machine Learning How-to](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml) to get started. +[Azure Machine Learning](https://docs.microsoft.com/en-us/azure/machine-learning/) enables you to train and deploy your machine learning models and pipelines at scale, and leverage open-source Python frameworks, such as PyTorch, TensorFlow, and scikit-learn. If you are looking at getting started with using the code in this repository with Azure Machine Learning, refer to [Azure Machine Learning How-to](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml) to get started. ## Interpretation For seismic interpretation, the repository consists of extensible machine learning pipelines, that shows how you can leverage state-of-the-art segmentation algorithms (UNet, SEResNET, HRNet) for seismic interpretation. @@ -113,7 +113,7 @@ data └── train_seismic.npy ``` -To prepare the data for the experiments (e.g. split into train/val/test), please run the following script: +To prepare the data for the experiments (e.g. split into train/val/test), first change `data_dir` definition to `data_dir="$HOME/data/dutch/data"` , and please run the following script: ``` # change working directory to scripts folder @@ -229,8 +229,8 @@ This section contains benchmarks of different algorithms for seismic interpretat #### Reproduce benchmarks -In order to reproduce the benchmarks, you will need to navigate to the [experiments](experiments) folder. In there, each of the experiments are split into different folders. To run the Netherlands F3 experiment navigate to the [dutchf3_patch/local](experiments/dutchf3_patch/local) folder. In there is a training script [([train.sh](experiments/dutchf3_patch/local/train.sh)) -which will run the training for any configuration you pass in. Once you have run the training you will need to run the [test.sh](experiments/dutchf3_patch/local/test.sh) script. Make sure you specify +In order to reproduce the benchmarks, you will need to navigate to the [experiments](experiments) folder. In there, each of the experiments are split into different folders. To run the Netherlands F3 experiment navigate to the [dutchf3_patch/local](experiments/interpretation/dutchf3_patch/local) folder. In there is a training script [([train.sh](experiments//interpretation/dutchf3_patch/local/train.sh)) +which will run the training for any configuration you pass in. Once you have run the training you will need to run the [test.sh](experiments/interpretation/dutchf3_patch/local/test.sh) script. Make sure you specify the path to the best performing model from your training run, either by passing it in as an argument or altering the YACS config file. ## Contributing diff --git a/examples/interpretation/README.md b/examples/interpretation/README.md index a70bb57f..ab791efb 100644 --- a/examples/interpretation/README.md +++ b/examples/interpretation/README.md @@ -1,3 +1,5 @@ -The folder contains notebook examples illustrating the use of segmentation algorithms on openly available datasets. Make sure you have followed the [set up instructions](../README.md) before running these examples. We provide the following notebook examples +The folder contains notebook examples illustrating the use of segmentation algorithms on openly available datasets. Make sure you have followed the [set up instructions](../../README.md) before running these examples. We provide the following notebook examples * [Dutch F3 dataset](notebooks/Dutch_F3_patch_model_training_and_evaluation.ipynb): This notebook illustrates section and patch based segmentation approaches on the [Dutch F3](https://terranubis.com/datainfo/Netherlands-Offshore-F3-Block-Complete) open dataset. This notebook uses denconvolution based segmentation algorithm on 2D patches. The notebook will guide you through visualization of the input volume, setting up model training and evaluation. + +To understand the configuration files and the dafault parameters refer to this [section in the top level README](../../README.md#configuration-files) \ No newline at end of file diff --git a/experiments/interpretation/dutchf3_patch/README.md b/experiments/interpretation/dutchf3_patch/README.md index 8fbd2d60..4b9ca4b2 100644 --- a/experiments/interpretation/dutchf3_patch/README.md +++ b/experiments/interpretation/dutchf3_patch/README.md @@ -4,8 +4,8 @@ You can run five different models on this dataset: * [HRNet](local/configs/hrnet.yaml) * [SEResNet](local/configs/seresnet_unet.yaml) * [UNet](local/configs/unet.yaml) -* [PatchDeconvNet](local/configs/patch_patch_deconvnet.yaml) -* [PatchDeconvNet-Skip](local/configs/patch_deconvnet_skip.yaml.yaml) +* [PatchDeconvNet](local/configs/patch_deconvnet.yaml) +* [PatchDeconvNet-Skip](local/configs/patch_deconvnet_skip.yaml) All these models take 2D patches of the dataset as input and provide predictions for those patches. The patches need to be stitched together to form a whole inline or crossline. From 799e7e6f2bc51f282f83213574c2897df64b5ed0 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Wed, 27 May 2020 20:03:49 +0000 Subject: [PATCH 2/9] README update --- README.md | 7 +++++-- docker/README.md | 4 ++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index d10b200d..c087347a 100644 --- a/README.md +++ b/README.md @@ -113,16 +113,19 @@ data └── train_seismic.npy ``` -To prepare the data for the experiments (e.g. split into train/val/test), first change `data_dir` definition to `data_dir="$HOME/data/dutch/data"` , and please run the following script: +To prepare the data for the experiments (e.g. split into train/val/test), please run the following script: ``` # change working directory to scripts folder cd scripts # For patch-based experiments -python prepare_dutchf3.py split_train_val patch --data_dir=${data_dir} --label_file=train/train_labels.npy --output_dir=splits \ +python prepare_dutchf3.py split_train_val patch --data_dir=${data_dir}/data --label_file=train/train_labels.npy --output_dir=splits \ --stride=50 --patch_size=100 --split_direction=both +# For section-based experiments +python prepare_dutchf3.py split_train_val section --data-dir=${data_dir}/data --label_file=train/train_labels.npy --output_dir=splits \ --split_direction=both + # go back to repo root cd .. ``` diff --git a/docker/README.md b/docker/README.md index 28cbba69..5bf761a3 100644 --- a/docker/README.md +++ b/docker/README.md @@ -2,13 +2,13 @@ This Docker image allows the user to run the notebooks in this repository on any # Download the HRNet model: -To run the [`HRNet_Penobscot_demo_notebook.ipynb`](https://github.com/microsoft/seismic-deeplearning/blob/master/examples/interpretation/notebooks/HRNet_Penobscot_demo_notebook.ipynb), you will need to manually download the [HRNet-W48-C](https://1drv.ms/u/s!Aus8VCZ_C_33dKvqI6pBZlifgJk) pretrained model. You can follow the instructions [here.](https://github.com/microsoft/seismic-deeplearning#hrnet). +To run the [`HRNet_Penobscot_demo_notebook.ipynb`](https://github.com/microsoft/seismic-deeplearning/blob/master/examples/interpretation/notebooks/HRNet_Penobscot_demo_notebook.ipynb), you will need to manually download the [HRNet-W48-C](https://1drv.ms/u/s!Aus8VCZ_C_33dKvqI6pBZlifgJk) pretrained model. You can follow the instructions [here.](../README.md#pretrained-models). If you are using an Azure Virtual Machine to run this code, you can download the model to your local machine, and then copy it to your Azure VM through the command below. Please make sure you update the `` and `` feilds. ```bash scp hrnetv2_w48_imagenet_pretrained.pth @:/home//seismic-deeplearning/docker/hrnetv2_w48_imagenet_pretrained.pth ``` -Once you have the model downloaded (ideally under the `docker` directory), you can process to build the Docker image. +Once you have the model downloaded (ideally under the `docker` directory), you can proceed to build the Docker image. # Build the Docker image: From 712dcad42803bfb690e5307f447cfaf66454f632 Mon Sep 17 00:00:00 2001 From: Fatemeh Zamanian Date: Thu, 28 May 2020 14:06:17 +0000 Subject: [PATCH 3/9] docker README update --- docker/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docker/README.md b/docker/README.md index 5bf761a3..ff0e27bf 100644 --- a/docker/README.md +++ b/docker/README.md @@ -1,8 +1,8 @@ -This Docker image allows the user to run the notebooks in this repository on any operating system without having to setup the environment or install anything other than the Docker engine. For instructions on how to install the Docker engine, click [here](https://www.docker.com/get-started). +This Docker image allows the user to run the notebooks in this repository on any unix based operating system without having to setup the environment or install anything other than the Docker engine. We recommend using [Azure Data Science Virtual Machine (DSVM) for Linux (Ubuntu)](https://docs.microsoft.com/en-us/azure/machine-learning/data-science-virtual-machine/dsvm-ubuntu-intro) as outlined [here](../README.md#compute-environment). For instructions on how to install the Docker engine, click [here](https://www.docker.com/get-started). # Download the HRNet model: -To run the [`HRNet_Penobscot_demo_notebook.ipynb`](https://github.com/microsoft/seismic-deeplearning/blob/master/examples/interpretation/notebooks/HRNet_Penobscot_demo_notebook.ipynb), you will need to manually download the [HRNet-W48-C](https://1drv.ms/u/s!Aus8VCZ_C_33dKvqI6pBZlifgJk) pretrained model. You can follow the instructions [here.](../README.md#pretrained-models). +To run the [`Dutch_F3_patch_model_training_and_evaluation.ipynb`](https://github.com/microsoft/seismic-deeplearning/blob/master/examples/interpretation/notebooks/Dutch_F3_patch_model_training_and_evaluation.ipynb), you will need to manually download the [HRNet-W48-C](https://1drv.ms/u/s!Aus8VCZ_C_33dKvqI6pBZlifgJk) pretrained model. You can follow the instructions [here.](../README.md#pretrained-models). If you are using an Azure Virtual Machine to run this code, you can download the model to your local machine, and then copy it to your Azure VM through the command below. Please make sure you update the `` and `` feilds. ```bash From f6e80f7013d4d230efac449e9330528c6bdbc544 Mon Sep 17 00:00:00 2001 From: Fatemeh Date: Thu, 28 May 2020 14:08:54 +0000 Subject: [PATCH 4/9] same --- docker/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker/README.md b/docker/README.md index ff0e27bf..f70ca0b3 100644 --- a/docker/README.md +++ b/docker/README.md @@ -1,4 +1,4 @@ -This Docker image allows the user to run the notebooks in this repository on any unix based operating system without having to setup the environment or install anything other than the Docker engine. We recommend using [Azure Data Science Virtual Machine (DSVM) for Linux (Ubuntu)](https://docs.microsoft.com/en-us/azure/machine-learning/data-science-virtual-machine/dsvm-ubuntu-intro) as outlined [here](../README.md#compute-environment). For instructions on how to install the Docker engine, click [here](https://www.docker.com/get-started). +This Docker image allows the user to run the notebooks in this repository on any Unix based operating system without having to setup the environment or install anything other than the Docker engine. We recommend using [Azure Data Science Virtual Machine (DSVM) for Linux (Ubuntu)](https://docs.microsoft.com/en-us/azure/machine-learning/data-science-virtual-machine/dsvm-ubuntu-intro) as outlined [here](../README.md#compute-environment). For instructions on how to install the Docker engine, click [here](https://www.docker.com/get-started). # Download the HRNet model: From c7468dc6a38026dc625487ead22e24bf00fe1fdb Mon Sep 17 00:00:00 2001 From: Fatemeh Date: Thu, 28 May 2020 17:07:34 +0000 Subject: [PATCH 5/9] readme update --- README.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/README.md b/README.md index c087347a..e36db79c 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,33 @@ This repository shows you how to perform seismic imaging and interpretation on Azure. It empowers geophysicists and data scientists to run seismic experiments using state-of-art DSL-based PDE solvers and segmentation algorithms on Azure. +Highlights of the latest release for Seismic Interpretation [v0.1.2](https://github.com/microsoft/seismic-deeplearning/tree/v0.1.2), + + +| Module | Features | +|-------------------------|------------------------------------------------------------------------------------------| +| Data ingestion pipeline | * Scripts to download and prepare Dutch F3 and Penobscot datasets. | +| | * Patch-based and section-based data prepration. | +| Training pipeline | * Configurable for different options through configuration files for five architectures: | +| | ---HRNet | +| | ---SEResNet | +| | ---UNet | +| | ---PatchDeconvNet | +| | ---PatchDeconvNet-Skip | +| | * Option to train a model from scratch. | +| | * Option to use a pre-trained a model. | +| Scoring pipeline | * Configurable to run for different training expriments. | + + +additionaly: + +|----------|------------| +| Docker | Enabled | +| OS | Linux | +| Hardware | Single GPU | + + + The repository provides sample notebooks, data loaders for seismic data, utilities, and out-of-the-box ML pipelines, organized as follows: - **sample notebooks**: these can be found in the `examples` folder - they are standard Jupyter notebooks which highlight how to use the codebase by walking the user through a set of pre-made examples - **experiments**: the goal is to provide runnable Python scripts that train and test (score) our machine learning models in the `experiments` folder. The models themselves are swappable, meaning a single train script can be used to run a different model on the same dataset by simply swapping out the configuration file which defines the model. From c5159d8f5131098f37f37a426e3288400fc13c4b Mon Sep 17 00:00:00 2001 From: Fatemeh Date: Thu, 28 May 2020 17:19:22 +0000 Subject: [PATCH 6/9] changes reverted --- README.md | 26 -------------------------- 1 file changed, 26 deletions(-) diff --git a/README.md b/README.md index e36db79c..7fc7c023 100644 --- a/README.md +++ b/README.md @@ -3,32 +3,6 @@ This repository shows you how to perform seismic imaging and interpretation on Azure. It empowers geophysicists and data scientists to run seismic experiments using state-of-art DSL-based PDE solvers and segmentation algorithms on Azure. -Highlights of the latest release for Seismic Interpretation [v0.1.2](https://github.com/microsoft/seismic-deeplearning/tree/v0.1.2), - - -| Module | Features | -|-------------------------|------------------------------------------------------------------------------------------| -| Data ingestion pipeline | * Scripts to download and prepare Dutch F3 and Penobscot datasets. | -| | * Patch-based and section-based data prepration. | -| Training pipeline | * Configurable for different options through configuration files for five architectures: | -| | ---HRNet | -| | ---SEResNet | -| | ---UNet | -| | ---PatchDeconvNet | -| | ---PatchDeconvNet-Skip | -| | * Option to train a model from scratch. | -| | * Option to use a pre-trained a model. | -| Scoring pipeline | * Configurable to run for different training expriments. | - - -additionaly: - -|----------|------------| -| Docker | Enabled | -| OS | Linux | -| Hardware | Single GPU | - - The repository provides sample notebooks, data loaders for seismic data, utilities, and out-of-the-box ML pipelines, organized as follows: - **sample notebooks**: these can be found in the `examples` folder - they are standard Jupyter notebooks which highlight how to use the codebase by walking the user through a set of pre-made examples From 24d49781d7afd8da75f3f868b114ba71de8a392a Mon Sep 17 00:00:00 2001 From: Fatemeh Date: Thu, 28 May 2020 22:15:08 +0000 Subject: [PATCH 7/9] typo fix --- README.md | 2 +- experiments/interpretation/dutchf3_patch/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7fc7c023..c9460565 100644 --- a/README.md +++ b/README.md @@ -233,7 +233,7 @@ This section contains benchmarks of different algorithms for seismic interpretat #### Reproduce benchmarks -In order to reproduce the benchmarks, you will need to navigate to the [experiments](experiments) folder. In there, each of the experiments are split into different folders. To run the Netherlands F3 experiment navigate to the [dutchf3_patch/local](experiments/interpretation/dutchf3_patch/local) folder. In there is a training script [([train.sh](experiments//interpretation/dutchf3_patch/local/train.sh)) +In order to reproduce the benchmarks, you will need to navigate to the [experiments](experiments) folder. In there, each of the experiments are split into different folders. To run the Netherlands F3 experiment navigate to the [dutchf3_patch/local](experiments/interpretation/dutchf3_patch/local) folder. In there is a training script [([train.sh](experiments/interpretation/dutchf3_patch/local/train.sh)) which will run the training for any configuration you pass in. Once you have run the training you will need to run the [test.sh](experiments/interpretation/dutchf3_patch/local/test.sh) script. Make sure you specify the path to the best performing model from your training run, either by passing it in as an argument or altering the YACS config file. diff --git a/experiments/interpretation/dutchf3_patch/README.md b/experiments/interpretation/dutchf3_patch/README.md index 4b9ca4b2..d8c98b40 100644 --- a/experiments/interpretation/dutchf3_patch/README.md +++ b/experiments/interpretation/dutchf3_patch/README.md @@ -18,7 +18,7 @@ Also follow instructions for [downloading and preparing](../../../README.md#f3-N ### Running experiments -Now you're all set to run training and testing experiments on the F3 Netherlands dataset. Please start from the `train.sh` and `test.sh` scripts under the `local/` and `distributed/` directories, which invoke the corresponding python scripts. Take a look at the project configurations in (e.g in `default.py`) for experiment options and modify if necessary. +Now you're all set to run training and testing experiments on the F3 Netherlands dataset. Please start from the `train.sh` and `test.sh` scripts under the `local/` directory, which invoke the corresponding python scripts. Take a look at the project configurations in (e.g in `default.py`) for experiment options and modify if necessary. ### Monitoring progress with TensorBoard - from the this directory, run `tensorboard --logdir='output'` (all runtime logging information is From 57f379e790e715db30de41cb8d4fc2288347818d Mon Sep 17 00:00:00 2001 From: Fatemeh Date: Thu, 28 May 2020 22:25:01 +0000 Subject: [PATCH 8/9] removing user specific path from config --- .../interpretation/dutchf3_patch/local/configs/hrnet.yaml | 2 +- .../dutchf3_patch/local/configs/patch_deconvnet.yaml | 2 +- .../dutchf3_patch/local/configs/seresnet_unet.yaml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/experiments/interpretation/dutchf3_patch/local/configs/hrnet.yaml b/experiments/interpretation/dutchf3_patch/local/configs/hrnet.yaml index 52263bbf..e6e5091a 100644 --- a/experiments/interpretation/dutchf3_patch/local/configs/hrnet.yaml +++ b/experiments/interpretation/dutchf3_patch/local/configs/hrnet.yaml @@ -93,7 +93,7 @@ VALIDATION: BATCH_SIZE_PER_GPU: 128 TEST: - MODEL_PATH: "/data/home/mat/repos/DeepSeismic/experiments/interpretation/dutchf3_patch/local/output/staging/0d1d2bbf9685995a0515ca1d9de90f9bcec0db90/seg_hrnet/Dec20_233535/models/seg_hrnet_running_model_33.pth" + MODEL_PATH: "/home/username/seismic-deeplearning/docker/hrnetv2_w48_imagenet_pretrained.pth" TEST_STRIDE: 10 SPLIT: 'Both' # Can be Both, Test1, Test2 INLINE: True diff --git a/experiments/interpretation/dutchf3_patch/local/configs/patch_deconvnet.yaml b/experiments/interpretation/dutchf3_patch/local/configs/patch_deconvnet.yaml index b5c59655..0d5ee793 100644 --- a/experiments/interpretation/dutchf3_patch/local/configs/patch_deconvnet.yaml +++ b/experiments/interpretation/dutchf3_patch/local/configs/patch_deconvnet.yaml @@ -47,7 +47,7 @@ VALIDATION: BATCH_SIZE_PER_GPU: 64 TEST: - MODEL_PATH: "/data/home/mat/repos/DeepSeismic/interpretation/experiments/segmentation/dutchf3/local/output/mat/exp/5cc37bbe5302e1989ef1388d629400a16f82d1a9/patch_deconvnet/Aug27_200339/models/patch_deconvnet_snapshot1model_50.pth" + MODEL_PATH: "" TEST_STRIDE: 10 SPLIT: 'Both' # Can be Both, Test1, Test2 INLINE: True diff --git a/experiments/interpretation/dutchf3_patch/local/configs/seresnet_unet.yaml b/experiments/interpretation/dutchf3_patch/local/configs/seresnet_unet.yaml index 81b4e54a..962fe8fe 100644 --- a/experiments/interpretation/dutchf3_patch/local/configs/seresnet_unet.yaml +++ b/experiments/interpretation/dutchf3_patch/local/configs/seresnet_unet.yaml @@ -49,7 +49,7 @@ VALIDATION: BATCH_SIZE_PER_GPU: 32 TEST: - MODEL_PATH: "/data/home/mat/repos/DeepSeismic/interpretation/experiments/segmentation/dutchf3/local/output/mat/exp/dc2e2d20b7f6d508beb779ffff37c77d0139e588/resnet_unet/Sep01_125513/models/resnet_unet_snapshot1model_52.pth" + MODEL_PATH: "" TEST_STRIDE: 10 SPLIT: 'Both' # Can be Both, Test1, Test2 INLINE: True From 0c11ac1dbe615a8b31773abff85d231ca17eda29 Mon Sep 17 00:00:00 2001 From: Fatemeh Date: Fri, 29 May 2020 21:17:28 +0000 Subject: [PATCH 9/9] minor fix --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c9460565..469149ba 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ This repository shows you how to perform seismic imaging and interpretation on A The repository provides sample notebooks, data loaders for seismic data, utilities, and out-of-the-box ML pipelines, organized as follows: - **sample notebooks**: these can be found in the `examples` folder - they are standard Jupyter notebooks which highlight how to use the codebase by walking the user through a set of pre-made examples - **experiments**: the goal is to provide runnable Python scripts that train and test (score) our machine learning models in the `experiments` folder. The models themselves are swappable, meaning a single train script can be used to run a different model on the same dataset by simply swapping out the configuration file which defines the model. -- **pip installable utilities**: we provide `cv_lib` and `deepseismic_interpretation` utilities (more info below) which are used by both sample notebooks and experiments mentioned above +- **pip installable utilities**: we provide `cv_lib` and `interpretation` utilities (more info below) which are used by both sample notebooks and experiments mentioned above DeepSeismic currently focuses on Seismic Interpretation (3D segmentation aka facies classification) with experimental code provided around Seismic Imaging in the contrib folder.