From 4518dd8f3820686ea11583d17349c90797c0a58e Mon Sep 17 00:00:00 2001 From: Eugene Fedorenko Date: Tue, 4 Feb 2020 13:55:33 -0800 Subject: [PATCH 1/5] typos --- README.md | 2 +- docs/getting_started.md | 9 ++++----- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index c0416cff..03b7ee8c 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ This reference architecture shows how to implement continuous integration (CI), 2. Once the Azure DevOps build pipeline is triggered, it performs code quality checks, data sanity tests, unit tests, builds an [Azure ML Pipeline](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-ml-pipelines) and publishes it in an [Azure ML Service Workspace](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace). 3. The [Azure ML Pipeline](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-ml-pipelines) is triggered once the Azure DevOps build pipeline completes. All the tasks in this pipeline runs on Azure ML Compute. Following are the tasks in this pipeline: - - **Train Model** task executes model training script on Azure ML Compute. It outputs a [model](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-azure-machine-learning-architecture#model) file which is stored in the [run history](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-azure-machine-learning-architecture#run). + - **Train Model** task executes model training script on Azure ML Compute. It outputs a [model](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-azure-machine-learning-architecture#models) file which is stored in the [run history](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-azure-machine-learning-architecture#runs). - **Evaluate Model** task evaluates the performance of the newly trained model with the model in production. If the new model performs better than the production model, the following steps are executed. If not, they will be skipped. diff --git a/docs/getting_started.md b/docs/getting_started.md index 3d372ab3..12477a05 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -67,7 +67,7 @@ be naming collisions with resources that require unique names like azure blob storage and registry DNS naming. Make sure to give a unique value to the BASE_NAME variable (e.g. MyUniqueML), so that the created resources will have unique names (e.g. MyUniqueMLamlcr, MyUniqueML-AML-KV, etc.). The length of -the BASE_NAME value should not exceed 10 characters. +the BASE_NAME value should not exceed 10 characters and should contain numbers and letters only. The **RESOURCE_GROUP** parameter is used as the name for the resource group that will hold the Azure resources for the solution. If providing an existing AML Workspace, set this value to the corresponding resource group name. @@ -116,11 +116,11 @@ Having done that, run the pipeline: ![iac run](./images/run-iac-pipeline.png) -Check out the newly created resources in the [Azure Portal](portal.azure.com): +Check out the newly created resources in the [Azure Portal](https://ms.portal.azure.com): ![created resources](./images/created-resources.png) -(Optional) To remove the resources created for this project you can use the [/environment_setup/iac-remove-environment.yml](../environment_setup/iac-remove-environment.yml) definition or you can just delete the resource group in the [Azure Portal](portal.azure.com). +(Optional) To remove the resources created for this project you can use the [/environment_setup/iac-remove-environment.yml](../environment_setup/iac-remove-environment.yml) definition or you can just delete the resource group in the [Azure Portal](https://ms.portal.azure.com). **Note:** The training ML pipeline uses a [sample diabetes dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_diabetes.html) as training data. If you want to use your own dataset, you need to [create and register a datastore](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-access-data#azure-machine-learning-studio) in your ML workspace and upload the datafile (e.g. [diabetes.csv](./data/diabetes.csv)) to the corresponding blob container. You can also define a datastore in the ML Workspace with [az cli](https://docs.microsoft.com/en-us/cli/azure/ext/azure-cli-ml/ml/datastore?view=azure-cli-latest#ext-azure-cli-ml-az-ml-datastore-attach-blob). You'll also need to configure DATASTORE_NAME and DATAFILE_NAME variables in ***devopsforai-aml-vg*** variable group. @@ -163,8 +163,7 @@ Once the pipeline is finished, explore the execution result: ![build](./images/multi-stage-aci.png) -and check out the published training pipeline in the **mlops-AML-WS** workspace in -[Azure Portal](https://ms.portal.azure.com/): +and check out the published training pipeline in the **mlops-AML-WS** workspace in [Azure Portal](https://ms.portal.azure.com/): ![training pipeline](./images/training-pipeline.png) From 4b3d419cb20f97cf3651900662dc8b51c5d63bb1 Mon Sep 17 00:00:00 2001 From: Eugene Fedorenko Date: Tue, 4 Feb 2020 13:57:41 -0800 Subject: [PATCH 2/5] typo --- =3.7 | 5 +++++ docs/getting_started.md | 2 +- environment_setup/Dockerfile | 1 + environment_setup/requirements.txt | 6 ++++-- 4 files changed, 11 insertions(+), 3 deletions(-) create mode 100644 =3.7 diff --git a/=3.7 b/=3.7 new file mode 100644 index 00000000..3fd85bb6 --- /dev/null +++ b/=3.7 @@ -0,0 +1,5 @@ +Requirement already up-to-date: flake8 in /usr/local/lib/python3.7/site-packages (3.7.9) +Requirement already satisfied, skipping upgrade: mccabe<0.7.0,>=0.6.0 in /Users/efedorenko/Library/Python/3.7/lib/python/site-packages (from flake8) (0.6.1) +Requirement already satisfied, skipping upgrade: pyflakes<2.2.0,>=2.1.0 in /Users/efedorenko/Library/Python/3.7/lib/python/site-packages (from flake8) (2.1.1) +Requirement already satisfied, skipping upgrade: entrypoints<0.4.0,>=0.3.0 in /Users/efedorenko/Library/Python/3.7/lib/python/site-packages (from flake8) (0.3) +Requirement already satisfied, skipping upgrade: pycodestyle<2.6.0,>=2.5.0 in /Users/efedorenko/Library/Python/3.7/lib/python/site-packages (from flake8) (2.5.0) diff --git a/docs/getting_started.md b/docs/getting_started.md index 12477a05..b4b11a41 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -67,7 +67,7 @@ be naming collisions with resources that require unique names like azure blob storage and registry DNS naming. Make sure to give a unique value to the BASE_NAME variable (e.g. MyUniqueML), so that the created resources will have unique names (e.g. MyUniqueMLamlcr, MyUniqueML-AML-KV, etc.). The length of -the BASE_NAME value should not exceed 10 characters and should contain numbers and letters only. +the BASE_NAME value should not exceed 10 characters and it should contain numbers and letters only. The **RESOURCE_GROUP** parameter is used as the name for the resource group that will hold the Azure resources for the solution. If providing an existing AML Workspace, set this value to the corresponding resource group name. diff --git a/environment_setup/Dockerfile b/environment_setup/Dockerfile index 5e7b7581..9c8c4b0d 100644 --- a/environment_setup/Dockerfile +++ b/environment_setup/Dockerfile @@ -10,5 +10,6 @@ COPY environment_setup/requirements.txt /setup/ RUN apt-get update && apt-get install gcc -y && pip install --upgrade -r /setup/requirements.txt && \ conda install -c r r-essentials +RUN chmod -R 777 /usr/local/lib/python3.7 CMD ["python"] \ No newline at end of file diff --git a/environment_setup/requirements.txt b/environment_setup/requirements.txt index f99e7f4b..b1a5c5b1 100644 --- a/environment_setup/requirements.txt +++ b/environment_setup/requirements.txt @@ -1,4 +1,5 @@ pytest>=5.3 +pytest-cov>=2.8.1 requests>=2.22 numpy>=1.17 pandas>=0.25 @@ -6,5 +7,6 @@ scikit-learn>=0.21.3 azureml-sdk>=1.0 python-dotenv>=0.10.3 flake8>=3.7 -flake8_formatter_junit_xml -azure-cli==2.0.76 +flake8_formatter_junit_xml>=0.0.6 +tox>=3.14.3 +azure-cli==2.0.76 \ No newline at end of file From 43b377d4b4bf375c4a4a7e04132f24340a819c17 Mon Sep 17 00:00:00 2001 From: Eugene Fedorenko Date: Tue, 4 Feb 2020 13:58:26 -0800 Subject: [PATCH 3/5] revert --- environment_setup/Dockerfile | 1 - 1 file changed, 1 deletion(-) diff --git a/environment_setup/Dockerfile b/environment_setup/Dockerfile index 9c8c4b0d..5e7b7581 100644 --- a/environment_setup/Dockerfile +++ b/environment_setup/Dockerfile @@ -10,6 +10,5 @@ COPY environment_setup/requirements.txt /setup/ RUN apt-get update && apt-get install gcc -y && pip install --upgrade -r /setup/requirements.txt && \ conda install -c r r-essentials -RUN chmod -R 777 /usr/local/lib/python3.7 CMD ["python"] \ No newline at end of file From 8d31544f6587387bed8947bff9cfa426fe4971dc Mon Sep 17 00:00:00 2001 From: Eugene Fedorenko Date: Tue, 4 Feb 2020 14:04:31 -0800 Subject: [PATCH 4/5] garbage --- =3.7 | 5 ----- 1 file changed, 5 deletions(-) delete mode 100644 =3.7 diff --git a/=3.7 b/=3.7 deleted file mode 100644 index 3fd85bb6..00000000 --- a/=3.7 +++ /dev/null @@ -1,5 +0,0 @@ -Requirement already up-to-date: flake8 in /usr/local/lib/python3.7/site-packages (3.7.9) -Requirement already satisfied, skipping upgrade: mccabe<0.7.0,>=0.6.0 in /Users/efedorenko/Library/Python/3.7/lib/python/site-packages (from flake8) (0.6.1) -Requirement already satisfied, skipping upgrade: pyflakes<2.2.0,>=2.1.0 in /Users/efedorenko/Library/Python/3.7/lib/python/site-packages (from flake8) (2.1.1) -Requirement already satisfied, skipping upgrade: entrypoints<0.4.0,>=0.3.0 in /Users/efedorenko/Library/Python/3.7/lib/python/site-packages (from flake8) (0.3) -Requirement already satisfied, skipping upgrade: pycodestyle<2.6.0,>=2.5.0 in /Users/efedorenko/Library/Python/3.7/lib/python/site-packages (from flake8) (2.5.0) From 95fc50c31a89f2fef76b72668c8be8e199ec7c2f Mon Sep 17 00:00:00 2001 From: Eugene Fedorenko Date: Wed, 5 Feb 2020 12:19:59 -0800 Subject: [PATCH 5/5] removed ms. --- docs/getting_started.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/getting_started.md b/docs/getting_started.md index 8657b27c..a5a6c669 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -115,11 +115,11 @@ Having done that, run the pipeline: ![iac run](./images/run-iac-pipeline.png) -Check out the newly created resources in the [Azure Portal](https://ms.portal.azure.com): +Check out the newly created resources in the [Azure Portal](https://portal.azure.com): ![created resources](./images/created-resources.png) -(Optional) To remove the resources created for this project you can use the [/environment_setup/iac-remove-environment.yml](../environment_setup/iac-remove-environment.yml) definition or you can just delete the resource group in the [Azure Portal](https://ms.portal.azure.com). +(Optional) To remove the resources created for this project you can use the [/environment_setup/iac-remove-environment.yml](../environment_setup/iac-remove-environment.yml) definition or you can just delete the resource group in the [Azure Portal](https://portal.azure.com). **Note:** The training ML pipeline uses a [sample diabetes dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_diabetes.html) as training data. If you want to use your own dataset, you need to [create and register a datastore](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-access-data#azure-machine-learning-studio) in your ML workspace and upload the datafile (e.g. [diabetes.csv](./data/diabetes.csv)) to the corresponding blob container. You can also define a datastore in the ML Workspace with [az cli](https://docs.microsoft.com/en-us/cli/azure/ext/azure-cli-ml/ml/datastore?view=azure-cli-latest#ext-azure-cli-ml-az-ml-datastore-attach-blob). You'll also need to configure DATASTORE_NAME and DATAFILE_NAME variables in ***devopsforai-aml-vg*** variable group. @@ -162,7 +162,7 @@ Once the pipeline is finished, explore the execution result: ![build](./images/multi-stage-aci.png) -and check out the published training pipeline in the **mlops-AML-WS** workspace in [Azure Portal](https://ms.portal.azure.com/): +and check out the published training pipeline in the **mlops-AML-WS** workspace in [Azure Portal](https://portal.azure.com/): ![training pipeline](./images/training-pipeline.png)