README.md

Data & AI Tech Immersion Workshop – Product Review Guide and Lab Instructions

AI, Experience 6 - MLOps with Azure Machine Learning and Azure DevOps

• Data & AI Tech Immersion Workshop – Product Review Guide and Lab Instructions
  • AI, Experience 6 - MLOps with Azure Machine Learning and Azure DevOps
  • Technology overview
  • Scenario overview
  • Exercise 1: Setup New Project in Azure DevOps
    • Task 1: Create New Project
    • Task 2: Import Quickstart code from a GitHub Repo
    • Task 3: Update the build YAML file
    • Task 4: Create new Service Connection
  • Exercise 2: Setup and Run the Build Pipeline
    • Task 1: Setup Build Pipeline
    • Task 2: Run the Build Pipeline
    • Task 3: Review Build Artifacts
    • Task 4: Review Build Outputs
  • Exercise 3: Setup the Release Pipeline
    • Task 1: Create an Empty Job
    • Task 2: Add Build Artifact
    • Task 3: Add Variables to Deploy & Test stage
    • Task 4: Setup Agent Pool for Deploy & Test stage
    • Task 5: Add Use Python Version task
    • Task 6: Add Install Requirements task
    • Task 7: Add Deploy & Test Webservice task
    • Task 8: Define Deployment Trigger
    • Task 9: Enable Continuous Deployment Trigger
    • Task 10: Save the Release Pipeline
  • Exercise 4: Test Build and Release Pipelines
    • Task 1: Make Edits to Source Code
    • Task 2: Monitor Build Pipeline
    • Task 3: Monitor Release Pipeline
    • Task 4: Review Release Pipeline Outputs
  • Wrap-up
  • Additional resources and more information

Technology overview

Azure Machine Learning uses a Machine Learning Operations (MLOps) approach, which improves the quality and consistency of your machine learning solutions. Azure Machine Learning Service provides the following MLOps capabilities:

• Integration with Azure Pipelines. Define continuous integration and deployment workflows for your models.
• A model registry that maintains multiple versions of your trained models.
• Model validation. Automatically validate your trained models and select the optimal configuration for deploying them into production.
• Deploy your models as a web service in the cloud, locally, or to IoT Edge devices.
• Monitor your deployed model's performance, so you can drive improvements in the next version of the model.

Scenario overview

In this experience you will learn how Contoso Auto can use MLOps to formalize the process of training and deploying new models using a DevOps approach.

Exercise 1: Setup New Project in Azure DevOps

Duration: 20 minutes

Task 1: Create New Project

1. Sign in to Azure DevOps.

2. Select New project.

   

3. Provide Project Name: mlops-quickstart and select Create.

   

Task 2: Import Quickstart code from a GitHub Repo

1. Within the new project:

   a. Select Repos from left navigation bar.

   b. Select Import from the content section.

   

2. Provide the following GitHub URL: https://github.com/solliancenet/mcw-mlops-starter and select Import. This should import the code required for the quickstart.

   

   Note that if you receive an error while importing the repository, please disable the preview feature New Repos landing pages and import the GitHub repository from the old UI, as shown in steps #3, #4, and #5 below.

3. [Optional] Select Account settings, Preview features.

   

4. [Optional] From the list of preview features, disable the preview feature New Repos landing pages.

   

5. [Optional] Repeat Step #1 above to import the GitHub repository from the old UI.

Task 3: Update the build YAML file

1. Select and open the azure-pipelines.yml file.

2. Select Edit and update the following variables: resourcegroup, and workspace. If you are using your own Azure subscription, please provide names to use. If an environment is provided to you, be sure to replace XXXXX in the values below with your unique identifier. Typically, for the provided environment, the format for the workspace is tech_immersion_aml_XXXXX where XXXXX is your unique identifier and edit the value of reource group to ti-XXXXX where XXXXX is your unique identifier.

   

3. Select Commit to save your changes.

   

Task 4: Create new Service Connection

1. From the left navigation select Project settings and then select Service connections.

   

2. Select Create service connection, select Azure Resource Manager, and then select Next.

   

3. If an environment is provided to you goto step #5. Select Service principal (automatic) and then select Next.

   

4. Provide the following information in the New Azure service connection dialog box and then select Save:

   a. Scope Level: Machine Learning Workspace

   b. Subscription: Select the Azure subscription to use.

   Note: It might take up to 30 seconds for the Subscription dropdown to be populated with available subscriptions, depending on the number of different subscriptions your account has access to.

   c. Resource group: This value should match the value you provided in the azure-pipelines.yml file. (Typically, for the provided environment, the format for the resourcegroup is tech_immersion_XXXXX.)

   d. Machine Learning Workspace: This value should match the value you provided in the azure-pipelines.yml file. (Typically, for the provided environment, the format for the workspace is tech_immersion_aml_XXXXX.)

   e. Service connection name: quick-starts-sc

   f. Grant access permission to all pipelines: this checkbox must be selected.

   

   Note: If you are unable to select your Machine Learning Workspace, do the following steps else skip to Exercise 2:

   • Quit the New Azure service connection dialog
   • Refresh or reload the web browser
   • Repeat steps 1-3 above
   • In step 4, change the Scope level to Subscription and then select your Resource group
   • Please remember to name your service connection as quick-starts-sc
   • Grant access permission to all pipelines

   Note: If you successfully created the new service connection goto Exercise 2.

5. Select Service principal (manual) and then select Next.

   

6. Provide the following information in the New Azure service connection dialog box and then select Verify and save:

   1. Scope Level: Subscription

   2. Subscription id: (Lab environment details page: Service Principal Details->Subscription Id)

   3. Subscription Name: You can find the subscription name from Azure Portal

   4. Service principal Id: (Lab environment details page: Service Principal Details->Application/Client Id)

   5. Service principal key: (Lab environment details page: Service Principal Details->Application Secret Key)

   6. Tenant ID: (Lab environment details page: Service Principal Details->Tenant Id)

   7. Service connection name: quick-starts-sc

   8. Grant access permission to all pipelines: this checkbox must be selected.

   

Exercise 2: Setup and Run the Build Pipeline

Duration: 25 minutes

Note: This exercise requires the new version of the Pipelines user interface. To activate it, select Pipelines from the left navigation. If the first option below Pipelines is Builds, you are still running on the previous version of the user interface. In this case, a popup should appear suggesting the activation of the new user interface.

Select Try it! to activate the new Pipelines user interface. When successfully activated, the first option below Pipelines from the left navigation will change to Pipelines.

Task 1: Setup Build Pipeline

1. From left navigation select Pipelines, Pipelines and then select Create pipeline.

   

2. Select Azure Repos Git as your code repository.

   

3. Select mlops-quickstart as your repository.

   

4. Review the YAML file.

   The build pipeline has four key steps:

   a. Attach folder to workspace and experiment. This command creates the .azureml subdirectory that contains a config.json file that is used to communicate with your Azure Machine Learning workspace. All subsequent steps rely on the config.json file to instantiate the workspace object.

   b. Create the AML Compute target to run your master pipeline for model training and model evaluation.

   c. Run the master pipeline. The master pipeline has two steps: (1) Train the machine learning model, and (2) Evaluate the trained machine learning model. The evaluation step evaluates if the new model performance is better than the currently deployed model. If the new model performance is improved, the evaluate step will create a new Image for deployment. The results of the evaluation step will be saved in a file called eval_info.json that will be made available for the release pipeline. You can review the code for the master pipeline and its steps in aml_service/pipelines_master.py, scripts/train.py, and scripts/evaluate.py.

   d. Publish the build artifacts. The snapshot of the repository, config.json, and eval_info.json files are published as build artifacts and thus can be made available for the release pipeline.

   

Task 2: Run the Build Pipeline

1. Select Run to start running your build pipeline.

   

2. Monitor the build run. The build pipeline, for the first run, will take around 20 minutes to run.

   

3. Select Job to monitor the detailed status of the build pipeline execution.

   

Task 3: Review Build Artifacts

1. The build will publish an artifact named devops-for-ai. Select Artifacts, 1 published to review the artifact contents.

   

2. Select outputs, eval_info.json and then select ellipsis and click on Download artifacts . The eval_info.json is the output from the model evaluation step and the information from the evaluation step will be later used in the release pipeline to deploy the model. Select the back arrow to return to the previous screen.

   

3. Open the eval_info.json in a json viewer or a text editor and observe the information. The json output contains information such as if the model passed the evaluation step (deploy_model: true or false) and evaluation accuracy.

   

Task 4: Review Build Outputs

1. Log in to Azure Machine Learning studio either directly or via the Azure Portal. Make sure you select the Azure Machine Learning workspace that you created from the notebook earlier and click on Get Started.

   

2. Open your Models section, and observe the versions of the registered model: compliance-classifier. The latest version is the one registered by the build pipeline you have run in the previous task.

   

3. Select the latest version of your model to review its properties. Notice the build_number tag which links the registered to model to the Azure DevOps build that generated it.

   

4. Open your Datasets section and observe the versions of the registered dataset: connected_car_components. The latest version is the one registered by the build pipeline you have run in the previous task.

   

5. Select the latest version of your dataset to review its properties. Notice the build_number tag that links the dataset version to the Azure DevOps build that generated it.

   

6. Select Models to view a list of registered models that reference the dataset.

   

Exercise 3: Setup the Release Pipeline

Duration: 20 minutes

Task 1: Create an Empty Job

1. Return to Azure DevOps and navigate to Pipelines, Releases and select New pipeline.

   

2. Select Empty job.

   

3. Provide Stage name: Deploy & Test and close the dialog.

   

Task 2: Add Build Artifact

1. Select Add an artifact.

   

2. Select Source type: Build, Source (build pipeline): mlops-quickstart. Observe the note that shows that the mlops-quickstart publishes the build artifact named devops-for-ai. Finally, select Add.

   

Task 3: Add Variables to Deploy & Test stage

1. Open View stage tasks link.

   

2. Open the Variables tab.

   

3. Add four Pipeline variables as name - value pairs and then select Save (use the default values in the Save dialog):

   a. Name: aks_name Value: aks-cluster01

   b. Name: aks_region Value: should be the same region as the region of your Azure Machine Learning workspace (e.g. eastus)

   c. Name: service_name Value: compliance-classifier-service

   d. Name: description Value: "Compliance Classifier Web Service" Note the double quotes around description value.

   Note:

   • Keep the scope for the variables to Deploy & Test stage.
   • The name of the Azure region should be the same one that was used to create Azure Machine Learning workspace earlier on.

   

Task 4: Setup Agent Pool for Deploy & Test stage

1. Open the Tasks tab.

   

2. Select Agent job and change Agent pool to Azure Pipelines and change Agent Specification to ubuntu-16.04.

   

Task 5: Add Use Python Version task

1. Select Add a task to Agent job (the + button), search for Use Python Version, and select Add.

   

2. Provide Display name: Use Python 3.6 and Version spec: 3.6.

   

Task 6: Add Install Requirements task

1. Select Add a task to Agent job (the + button), search for Bash, and select Add.

   

2. Provide Display name: Install Requirements and select object browser ... to provide Script Path.

   

3. Navigate to Linked artifacts/_mlops-quickstart (Build)/devops-for-ai/environment_setup and select install_requirements.sh.

   

4. Expand Advanced and select object browser ... to provide Working Directory.

   

5. Navigate to Linked artifacts/_mlops-quickstart (Build)/devops-for-ai and select environment_setup.

   

Task 7: Add Deploy & Test Webservice task

1. Select Add a task to Agent job (the + button), search for Azure CLI, and select Add.

   

2. Provide the following information for the Azure CLI task:

   a. Task version: 1.*

   b. Display name: Deploy and Test Webservice

   c. Azure subscription: quick-starts-sc

   Note: This is the service connection we created in Exercise 1 / Task 4.

   d. Script Location: Inline script

   e. Inline Script: python aml_service/deploy.py --service_name $(service_name) --aks_name $(aks_name) --aks_region $(aks_region) --description $(description)

   

3. Expand Advanced and provide Working Directory: $(System.DefaultWorkingDirectory)/_mlops-quickstart/devops-for-ai.

   

In a separate browser tab, navigate to the Repo section in Azure DevOps and please review the code in aml_service/deploy.py. This step will read the eval_info.json and if the evaluation step recommended to deploy the new trained model, it will deploy the new model to production in an Azure Kubernetes Service (AKS) cluster.

Task 8: Define Deployment Trigger

1. Navigate to Pipeline tab, and select Pre-deployment conditions for the Deploy & Test stage.

2. Select After release.

   

3. Close the dialog.

Task 9: Enable Continuous Deployment Trigger

1. Select Continuous deployment trigger for _mlops-quickstart artifact.

2. Enable: Creates a release every time a new build is available.

   

3. Close the dialog

Task 10: Save the Release Pipeline

1. Provide name: mlops-quickstart-release.

2. Select: Save (use the default values in the Save dialog).

   

Exercise 4: Test Build and Release Pipelines

Duration: 30 minutes

Task 1: Make Edits to Source Code

1. Navigate to: Repos -> Files -> scripts -> train.py.

2. Edit train.py.

3. Change the learning rate (lr) for the optimizer from 0.1 to 0.001.

4. Change the number of training epochs from 3 to 5.

5. Select Commit.

   

6. Provide comment: Improving model performance: changed learning rate. and select Commit.

   

Task 2: Monitor Build Pipeline

1. Navigate to Pipelines, Pipelines. Observe that the CI build is triggered because of the source code change.

   

2. Select the pipeline run and monitor the pipeline steps. The pipeline will run for 16-18 minutes. Proceed to the next task when the build pipeline successfully completes.

   

Task 3: Monitor Release Pipeline

1. Navigate to Pipelines, Releases. Observe that the Release pipeline is automatically trigger upon successful completion of the build pipeline. Select as shown in the figure to view pipeline logs.

   

2. The release pipeline will run for about 15 minutes. Proceed to the next task when the release pipeline successfully completes.

Task 4: Review Release Pipeline Outputs

1. From the pipeline logs view, select Deploy & Test Webservice task to view details.

   

2. Observe the Scoring URI and API Key for the deployed webservice. Please note down both the Scoring URI and API Key for Exercise 7.

   

3. Log in to Azure Machine Learning studio. Open your Endpoints section, and observe the deployed webservice: compliance-classifier-service.

   

Wrap-up

Congratulations on completing this experience.

To recap, you experienced:

1. Creating a new project in Azure DevOps.

2. Creating a Build Pipeline to support model training.

3. Creating a Release Pipeline to support model deployment.

Additional resources and more information

To learn more about MLOps with the Azure Machine Learning service, visit the documentation