Cloud Pak for Data/WML/notebooks/model risk management/OpenScale MRM Cloud pak for Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "968913f3-8cb5-42a3-99d0-a40ef9b6f18a"
   },
   "source": [
    "<img src=\"https://github.com/pmservice/ai-openscale-tutorials/raw/master/notebooks/images/banner.png\" align=\"left\" alt=\"banner\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4f7234e8-ee83-41dd-9596-89b0b50274fe"
   },
   "source": [
    "This notebook should be run using **Default Spark 3.3.x & Python 3.10** or **Python 3.9**  runtime environment. If you are viewing this in Watson Studio and do not see Python 3.10.x with Spark 3.3.x in the upper right corner of your screen, please update the runtime now. It requires service credentials for the following services:\n",
    "\n",
    "<li>Watson OpenScale\n",
    "<li>Watson Machine Learning\n",
    "<li>Db2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8a116e3e-792c-47a3-b287-e0c69d8f0cbe"
   },
   "source": [
    "## Credentials for IBM Cloud Pak for Data\n",
    "In the following code boxes, you need to replace the sample data with your own credentials. You can acquire the information from your system administrator or through the Cloud Pak for Data dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "959689e776dd465089bf47b9876ea020"
   },
   "source": [
    "## Package installation\n",
    "The following opensource packages must be installed into this notebook instance so that they are available to use during processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "e4753d2f0a324dafa328c294c104ad04"
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "402353099aa043fd8a602c6f8fbb11bc"
   },
   "outputs": [],
   "source": [
    "#If you are running this notebook in non IBM Watson Studio env then uncomment the below pip statements and run it\n",
    "#!pip install --upgrade pyspark==3.2.0 --no-cache | tail -n 1\n",
    "#!pip install --upgrade pandas==1.2.3 --no-cache | tail -n 1\n",
    "#!pip install --upgrade requests==2.23 --no-cache | tail -n 1\n",
    "#!pip install numpy==1.20.1 --no-cache | tail -n 1\n",
    "#!pip install scikit-learn==1.1 --no-cache | tail -n 1\n",
    "#!pip install SciPy --no-cache | tail -n 1\n",
    "#!pip install lime --no-cache | tail -n 1\n",
    "\n",
    "!pip install --upgrade ibm-watson-machine-learning --user | tail -n 1\n",
    "!pip install --upgrade ibm-watson-openscale --no-cache | tail -n 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#If you runtime env doesn't have pyspark then please uncomment the below pip statement and run it.\n",
    "!pip install --upgrade pyspark==3.3 --no-cache | tail -n 1\n",
    "!pip install scikit-learn==1.1.0 --no-cache | tail -n 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "39a3057948634c6e974446488ed7cbf8"
   },
   "source": [
    "### Action: restart the kernel!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b91dea91-2604-47fc-897f-ead599a28538"
   },
   "source": [
    "### Obtaining your Watson OpenScale credentials\n",
    "You can retrieve the URL by running the following command: `oc get route -n namespace1 --no-headers | awk '{print $2}'`\n",
    "Replace the `namespace1` variable with your namespace.\n",
    "\n",
    "You should have been assigned a username and password when you were added to the Cloud Pak for Data system. You might need to ask either your database administrator or your system administrator for some of the information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "ea185c7c-00e5-4217-871e-2932711a5a89"
   },
   "outputs": [],
   "source": [
    "############################################################################################\n",
    "# Paste your Watson OpenScale credentials into the following section and then run this cell.\n",
    "############################################################################################\n",
    "\n",
    "WOS_CREDENTIALS = {\n",
    "     \"url\": \"https://********\",\n",
    "     \"username\": \"********\",\n",
    "     \"password\": \"********\"\n",
    " }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "62a99f39-d6c4-4b3c-8629-829a71f226bd"
   },
   "source": [
    "### Your Watson OpenScale GUID\n",
    "For most systems, the default GUID is already entered for you. You would only need to update this particular entry if the GUID was changed from the default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "febe1223-e2c8-4087-9447-b04435c6ac7f"
   },
   "outputs": [],
   "source": [
    "WOS_GUID=\"00000000-0000-0000-0000-000000000000\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3d4a05b2-7a03-46af-b833-e38c8f5ce433"
   },
   "source": [
    "### Your Watson OpenScale GUID\n",
    "For most systems, the default GUID is already entered for you. You would only need to update this particular entry if the GUID was changed from the default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "b4b6654e-7ae0-4831-8728-422c2ede1de0"
   },
   "outputs": [],
   "source": [
    "WML_CREDENTIALS = WOS_CREDENTIALS.copy()\n",
    "WML_CREDENTIALS['instance_id']='openshift'\n",
    "WML_CREDENTIALS['version']='4.7' #If your env is CP4D 4.0 then specify \"4.0\" instead of \"4.7\"  \n",
    "WML_CREDENTIALS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2f472554-d55b-4314-addb-34dd6c23ddb6"
   },
   "source": [
    "### Your database credentials and schema\n",
    "Normally, you must obtain the database credentials and schema name from your database administrator, however, if you deployed one of the database options from the Cloud Pak for Data UI, you should be able to retrieve credentials yourself by going to My data and right-clicking on the database tile."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "f17f892f-1d20-4171-a9e4-e2f9bd2e9498"
   },
   "outputs": [],
   "source": [
    "####################################################################################\n",
    "# Paste your database credentials into the following section and then run this cell.\n",
    "####################################################################################\n",
    "# DATABASE_CREDENTIALS = {\n",
    "#     \"db\": \"SAMPLE\",\n",
    "#     \"db_type\": \"db2\",\n",
    "#     \"hostname\": \"******\",\n",
    "#     \"password\": \"******\",\n",
    "#     \"port\": ******,\n",
    "#     \"username\": \"******\"\n",
    "# }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "130f9cab-b28e-4387-a050-bc419ec7b639"
   },
   "outputs": [],
   "source": [
    "SCHEMA_NAME = \"YOUR_SCHEMA_NAME_HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c3c3c3f2-34a2-4d8b-9a8e-0ba26b56eaeb"
   },
   "source": [
    "## Load the training data from Github\n",
    "So you don't have to manually generate training data, we've provided a sample and placed it in a publicly available Github repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "b7baf81e-49e9-4658-ae58-43a8d494dd94"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-08-07 11:28:25--  https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/german_credit_data_biased_training.csv\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.110.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 689622 (673K) [text/plain]\n",
      "Saving to: ‘german_credit_data_biased_training.csv’\n",
      "\n",
      "german_credit_data_ 100%[===================>] 673.46K  --.-KB/s    in 0.1s    \n",
      "\n",
      "2024-08-07 11:28:25 (6.27 MB/s) - ‘german_credit_data_biased_training.csv’ saved [689622/689622]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm german_credit_data_biased_training.csv\n",
    "!wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/german_credit_data_biased_training.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "78fdc41f-fedd-4c67-a5a6-6369ece571c5"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import requests\n",
    "import base64\n",
    "from requests.auth import HTTPBasicAuth\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "59d18c2d-87b6-40b9-83d1-7d5cdb7f8738"
   },
   "source": [
    "# Deploy the models\n",
    "\n",
    "The following cells deploy both the pre-prod and challenger models into the Watson Machine Learning instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "3a9a92db-3283-4a32-bbdd-efd1d3229733"
   },
   "outputs": [],
   "source": [
    "PRE_PROD_MODEL_NAME=\"German Credit Risk Model - PreProd\"\n",
    "PRE_PROD_DEPLOYMENT_NAME=\"German Credit Risk Model - PreProd\"\n",
    "\n",
    "PRE_PROD_CHALLENGER_MODEL_NAME=\"German Credit Risk Model - Challenger\"\n",
    "PRE_PROD_CHALLENGER_DEPLOYMENT_NAME=\"German Credit Risk Model - Challenger\"\n",
    "\n",
    "PRE_PROD_SPACE_NAME=\"wml_preprod\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "33f8c143-b900-4616-9f5b-287891dde7ea"
   },
   "source": [
    "## Initialize WML client and set the space_id parameter\n",
    "Analytic spaces are used to categorize assets and can be associated with a project inside of Watson Studio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "6ab80789-141f-4404-9f27-c1097a979edf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://cpd-cpd-instance.apps.wos415nfs2672.cp.fyre.ibm.com\n",
      "Note: 'limit' is not provided. Only first 50 records will be displayed if the number of records exceed 50\n",
      "------------------------------------  -----------------------------------------------------------------------  ------------------------\n",
      "ID                                    NAME                                                                     CREATED\n",
      "06873ce9-baf3-46a6-86c4-54c1e3d84429  tutorial_space                                                           2024-08-06T13:29:18.489Z\n",
      "e8b87647-a0e9-4932-920f-1e8f1d1f383d  tutorial-space                                                           2024-08-06T12:55:07.962Z\n",
      "832d6bbe-a2ea-451f-8d94-853da7c52b00  FrauDScore                                                               2024-08-05T07:48:21.490Z\n",
      "8d3a3e65-4875-4a5e-a6bd-617d2deeb9ac  Neelima                                                                  2024-08-05T05:42:10.490Z\n",
      "3a051b57-d5f6-4290-a2ac-5f66a6ed6d65  openscale-express-path-preprod-00000000-0000-0000-0000-1722503677398532  2024-08-01T09:56:04.456Z\n",
      "3d10da0f-497d-43d6-a8cd-71d15b833511  openscale-express-path-00000000-0000-0000-0000-1722503677398532          2024-08-01T09:55:43.160Z\n",
      "45683c18-10ea-4b25-8888-cf3c12646b8d  hk_pre_prod                                                              2024-07-27T12:34:07.223Z\n",
      "7f4afba9-1a6b-4f27-9a0a-507182adf269  goSales deployment space                                                 2024-07-09T14:42:26.982Z\n",
      "47dab2f7-5fe2-49bb-bf69-2cad9c7c5aa8  Notebook demo space                                                      2024-07-09T07:44:12.138Z\n",
      "a4bc50f8-8acf-47b5-b82d-103f311a5fa4  GCR Auto AI prod space                                                   2024-07-04T06:40:01.094Z\n",
      "4a0ad576-aad5-43ee-9457-11b4d2f39cf6  GCR AutoAI space Demo                                                    2024-07-03T14:26:17.440Z\n",
      "a8233d23-2f8b-40fc-b9a2-45aec8b53b83  openscale-express-path-preprod-00000000-0000-0000-0000-1719914010257861  2024-07-02T09:59:03.652Z\n",
      "3e3e2ae1-23d9-468d-abff-02c127d052ef  openscale-express-path-00000000-0000-0000-0000-1719914010257861          2024-07-02T09:58:32.157Z\n",
      "dcb2cb00-cb6f-47fe-8543-cffd5df0786a  notebooks CPD                                                            2024-07-01T17:10:55.025Z\n",
      "c3905865-4e73-43d1-a18f-24cf3a8b9163  wml_preprod                                                              2024-06-29T17:49:53.207Z\n",
      "7e5a8be6-9103-4c22-9c43-b66f3d8364de  poojitha_notebooks_space                                                 2024-06-29T13:53:46.645Z\n",
      "16ccd855-46bd-43ed-8219-5f00ac565d08  shreya-space                                                             2024-06-26T04:29:17.302Z\n",
      "bc3b9797-c509-4fb4-a424-f67b1e2ed4be  QUALITY_WMLV4_PREPROD                                                    2024-06-23T12:23:04.790Z\n",
      "e396e187-2977-47b4-ade3-1539f9f10adc  QUALITY_WMLV4_PROD                                                       2024-06-23T12:22:54.422Z\n",
      "40c4d032-0339-4da6-bfec-4bdb096c9650  shreya                                                                   2024-06-20T10:54:20.088Z\n",
      "088c142e-f35e-4e48-a30c-ad55a6edeecc  notebooks 5.0                                                            2024-06-13T04:42:07.336Z\n",
      "b9b3d3b4-6e26-4e16-807d-e8bf5e7d6984  MRM_WMLV4_PREPROD                                                        2024-06-12T15:49:26.571Z\n",
      "d22e2b6b-917c-4427-a40c-1a439352a742  MRM_WMLV4_PROD                                                           2024-06-12T15:49:16.185Z\n",
      "ce15e0f6-be30-4349-af47-35ae15983bf1  openscale-express-path-preprod-00000000-0000-0000-0000-000000000000      2024-06-04T05:18:51.988Z\n",
      "6264dc0e-087a-4dea-bcbc-6bd872b510fb  openscale-express-path-00000000-0000-0000-0000-000000000000              2024-06-04T05:18:30.811Z\n",
      "------------------------------------  -----------------------------------------------------------------------  ------------------------\n",
      "c3905865-4e73-43d1-a18f-24cf3a8b9163\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'SUCCESS'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from ibm_watson_machine_learning import APIClient\n",
    "wml_client = APIClient(WML_CREDENTIALS)\n",
    "print(wml_client.service_instance.get_url())\n",
    "\n",
    "wml_client.spaces.list()\n",
    "\n",
    "# Find and set the default space\n",
    "space_name=PRE_PROD_SPACE_NAME\n",
    "spaces = wml_client.spaces.get_details()['resources']\n",
    "space_id = None\n",
    "for space in spaces:\n",
    "    if space['entity']['name'] == space_name:\n",
    "        space_id = space[\"metadata\"][\"id\"]\n",
    "        print(space_id)\n",
    "if space_id is None:\n",
    "    space_id = wml_client.spaces.store(\n",
    "        meta_props={wml_client.spaces.ConfigurationMetaNames.NAME: space_name})[\"metadata\"][\"id\"]\n",
    "\n",
    "wml_client.set.default_space(space_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "35c292be1eee4a87806e83bfa8c96500"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.0.360'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wml_client.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cb16f23a-8466-4b34-a264-837ea7889d95"
   },
   "source": [
    "## Deploy the Spark Credit Risk Model to WML\n",
    "\n",
    "The following cell deploys the Spark version of the German Credit Risk Model to the specified Machine Learning instance in the specified deployment space. You'll notice that this version of the German Credit Risk model has an auc-roc score around 71%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "76334df0-8942-4ba5-9798-7d0d33990741"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/08/07 11:28:44 WARN Utils: Your hostname, Nelwins-MacBook-Pro.local resolves to a loopback address: 127.0.0.1; using 192.168.0.103 instead (on interface en0)\n",
      "24/08/07 11:28:44 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n",
      "24/08/07 11:28:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of records for training: 4485\n",
      "Number of records for evaluation: 515\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/08/07 11:28:55 WARN InstanceBuilder: Failed to load implementation from:dev.ludovic.netlib.blas.JNIBLAS\n",
      "24/08/07 11:28:55 WARN InstanceBuilder: Failed to load implementation from:dev.ludovic.netlib.blas.VectorBLAS\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy = 0.726321\n",
      "Deleting deployment id 9190b59f-516f-4b54-8404-6e72e94ef9fe\n",
      "Deleting model id 044c642b-7516-4edf-b5e6-744a13dad199\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/08/07 11:28:58 WARN GarbageCollectionMetrics: To enable non-built-in garbage collector(s) List(G1 Concurrent GC), users should configure it(them) to spark.eventLog.gcMetrics.youngGenerationGarbageCollectors or spark.eventLog.gcMetrics.oldGenerationGarbageCollectors\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------  -------------------------------------  ------------------------  ---------  ----------  ----------------\n",
      "ID                                    NAME                                   CREATED                   TYPE       SPEC_STATE  SPEC_REPLACEMENT\n",
      "ba51f79c-34b7-4f66-b50b-a55dee0bb14c  German Credit Risk Model - Challenger  2024-08-07T05:51:33.002Z  mllib_3.3  deprecated  spark-mllib_3.4\n",
      "------------------------------------  -------------------------------------  ------------------------  ---------  ----------  ----------------\n",
      "{'name': 'German Credit Risk Model - PreProd', 'description': 'German Credit Risk Model - PreProd', 'software_spec': 'd11f2434-4fc7-58b7-8a62-755da64fdaf8', 'type': 'mllib_3.3'}\n",
      "Note: Warnings!! :  Software specification spark-mllib_3.3 specified for the wml_model is deprecated and will be removed in the future. We recommend you use spark-mllib_3.4 instead. For details see Supported Frameworks https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=specifications-supported-deployment-frameworks\n",
      "{'entity': {'hybrid_pipeline_software_specs': [], 'label_column': 'Risk', 'pipeline': {'id': '68fe2bd1-0132-4212-85a0-d6dacef564ef'}, 'schemas': {'input': [{'fields': [{'metadata': {}, 'name': 'CheckingStatus', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'LoanDuration', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'CreditHistory', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'LoanPurpose', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'LoanAmount', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'ExistingSavings', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'EmploymentDuration', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'InstallmentPercent', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'Sex', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'OthersOnLoan', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'CurrentResidenceDuration', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'OwnsProperty', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'Age', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'InstallmentPlans', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'Housing', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'ExistingCreditsCount', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'Job', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'Dependents', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'Telephone', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'ForeignWorker', 'nullable': True, 'type': 'string'}], 'id': '1', 'type': 'struct'}], 'output': []}, 'software_spec': {'id': 'd11f2434-4fc7-58b7-8a62-755da64fdaf8', 'name': 'spark-mllib_3.3'}, 'training_data_references': [{'id': '1', 'location': {}, 'schema': {'fields': [{'metadata': {}, 'name': 'CheckingStatus', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'LoanDuration', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'CreditHistory', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'LoanPurpose', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'LoanAmount', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'ExistingSavings', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'EmploymentDuration', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'InstallmentPercent', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'Sex', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'OthersOnLoan', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'CurrentResidenceDuration', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'OwnsProperty', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'Age', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'InstallmentPlans', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'Housing', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'ExistingCreditsCount', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'Job', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'Dependents', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'Telephone', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'ForeignWorker', 'nullable': True, 'type': 'string'}, {'metadata': {'modeling_role': 'target'}, 'name': 'Risk', 'nullable': True, 'type': 'string'}], 'id': '1', 'type': 'struct'}, 'type': 'fs'}], 'type': 'mllib_3.3'}, 'metadata': {'created_at': '2024-08-07T05:59:14.241Z', 'description': 'German Credit Risk Model - PreProd', 'id': 'f1b4fe35-1a75-4dca-a9b3-bc5dd7ed43db', 'modified_at': '2024-08-07T05:59:22.805Z', 'name': 'German Credit Risk Model - PreProd', 'owner': '1000331001', 'resource_key': '44a5d259-9ede-4fe2-bcd5-d5c8f7aba30a', 'space_id': 'c3905865-4e73-43d1-a18f-24cf3a8b9163'}, 'system': {'warnings': [{'message': 'Software specification spark-mllib_3.3 specified for the wml_model is deprecated and will be removed in the future. We recommend you use spark-mllib_3.4 instead. For details see Supported Frameworks https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=specifications-supported-deployment-frameworks'}]}}\n",
      "------------------------------------  -------------------------------------  ------------------------  ---------  ----------  ----------------\n",
      "ID                                    NAME                                   CREATED                   TYPE       SPEC_STATE  SPEC_REPLACEMENT\n",
      "f1b4fe35-1a75-4dca-a9b3-bc5dd7ed43db  German Credit Risk Model - PreProd     2024-08-07T05:59:14.002Z  mllib_3.3  deprecated  spark-mllib_3.4\n",
      "ba51f79c-34b7-4f66-b50b-a55dee0bb14c  German Credit Risk Model - Challenger  2024-08-07T05:51:33.002Z  mllib_3.3  deprecated  spark-mllib_3.4\n",
      "------------------------------------  -------------------------------------  ------------------------  ---------  ----------  ----------------\n",
      "Deploying model...\n",
      "\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "Synchronous deployment creation for uid: 'f1b4fe35-1a75-4dca-a9b3-bc5dd7ed43db' started\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "\n",
      "initializing\n",
      "Note: Software specification spark-mllib_3.3 is deprecated. Use spark-mllib_3.4 software specification instead when saving a spark model. For details, see https://www.ibm.com/support/producthub/icpdata/docs/content/SSQNUZ_latest/wsj/wmls/wmls-deploy-python-types.html.\n",
      "\n",
      "ready\n",
      "\n",
      "\n",
      "------------------------------------------------------------------------------------------------\n",
      "Successfully finished deployment creation, deployment_uid='42b13344-baa1-45a0-89ca-ef662387f691'\n",
      "------------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "Model id: f1b4fe35-1a75-4dca-a9b3-bc5dd7ed43db\n",
      "Deployment id: 42b13344-baa1-45a0-89ca-ef662387f691\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'predictions': [{'fields': ['CheckingStatus',\n",
       "    'LoanDuration',\n",
       "    'CreditHistory',\n",
       "    'LoanPurpose',\n",
       "    'LoanAmount',\n",
       "    'ExistingSavings',\n",
       "    'EmploymentDuration',\n",
       "    'InstallmentPercent',\n",
       "    'Sex',\n",
       "    'OthersOnLoan',\n",
       "    'CurrentResidenceDuration',\n",
       "    'OwnsProperty',\n",
       "    'Age',\n",
       "    'InstallmentPlans',\n",
       "    'Housing',\n",
       "    'ExistingCreditsCount',\n",
       "    'Job',\n",
       "    'Dependents',\n",
       "    'Telephone',\n",
       "    'ForeignWorker',\n",
       "    'CheckingStatus_IX',\n",
       "    'CreditHistory_IX',\n",
       "    'EmploymentDuration_IX',\n",
       "    'ExistingSavings_IX',\n",
       "    'ForeignWorker_IX',\n",
       "    'Housing_IX',\n",
       "    'InstallmentPlans_IX',\n",
       "    'Job_IX',\n",
       "    'LoanPurpose_IX',\n",
       "    'OthersOnLoan_IX',\n",
       "    'OwnsProperty_IX',\n",
       "    'Sex_IX',\n",
       "    'Telephone_IX',\n",
       "    'features',\n",
       "    'rawPrediction',\n",
       "    'probability',\n",
       "    'prediction',\n",
       "    'predictedLabel'],\n",
       "   'values': [['no_checking',\n",
       "     13,\n",
       "     'credits_paid_to_date',\n",
       "     'car_new',\n",
       "     1343,\n",
       "     '100_to_500',\n",
       "     '1_to_4',\n",
       "     2,\n",
       "     'female',\n",
       "     'none',\n",
       "     3,\n",
       "     'savings_insurance',\n",
       "     46,\n",
       "     'none',\n",
       "     'own',\n",
       "     2,\n",
       "     'skilled',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     [21,\n",
       "      [1, 3, 5, 13, 14, 15, 16, 17, 18, 19, 20],\n",
       "      [1.0, 1.0, 1.0, 13.0, 1343.0, 2.0, 3.0, 13.0, 46.0, 2.0, 1.0]],\n",
       "     [0.37923501094948475, -0.37923501094948475],\n",
       "     [0.6810214668555962, 0.31897853314440383],\n",
       "     0.0,\n",
       "     'No Risk'],\n",
       "    ['no_checking',\n",
       "     24,\n",
       "     'prior_payments_delayed',\n",
       "     'furniture',\n",
       "     4567,\n",
       "     '500_to_1000',\n",
       "     '1_to_4',\n",
       "     4,\n",
       "     'male',\n",
       "     'none',\n",
       "     4,\n",
       "     'savings_insurance',\n",
       "     36,\n",
       "     'none',\n",
       "     'free',\n",
       "     2,\n",
       "     'management_self-employed',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     [21,\n",
       "      [2, 3, 9, 10, 13, 14, 15, 16, 17, 18, 19, 20],\n",
       "      [1.0, 2.0, 2.0, 2.0, 24.0, 4567.0, 4.0, 4.0, 24.0, 36.0, 2.0, 1.0]],\n",
       "     [0.40678948974997703, -0.40678948974997703],\n",
       "     [0.6928716383154149, 0.3071283616845851],\n",
       "     0.0,\n",
       "     'No Risk'],\n",
       "    ['0_to_200',\n",
       "     26,\n",
       "     'all_credits_paid_back',\n",
       "     'car_new',\n",
       "     863,\n",
       "     'less_100',\n",
       "     'less_1',\n",
       "     2,\n",
       "     'female',\n",
       "     'co-applicant',\n",
       "     2,\n",
       "     'real_estate',\n",
       "     38,\n",
       "     'none',\n",
       "     'own',\n",
       "     1,\n",
       "     'skilled',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     2.0,\n",
       "     3.0,\n",
       "     3.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     2.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     [2.0,\n",
       "      3.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      3.0,\n",
       "      1.0,\n",
       "      1.0,\n",
       "      2.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      26.0,\n",
       "      863.0,\n",
       "      2.0,\n",
       "      2.0,\n",
       "      26.0,\n",
       "      38.0,\n",
       "      1.0,\n",
       "      1.0],\n",
       "     [1.08446701248213, -1.08446701248213],\n",
       "     [0.8974248814242887, 0.10257511857571133],\n",
       "     0.0,\n",
       "     'No Risk'],\n",
       "    ['0_to_200',\n",
       "     14,\n",
       "     'no_credits',\n",
       "     'car_new',\n",
       "     2368,\n",
       "     'less_100',\n",
       "     '1_to_4',\n",
       "     3,\n",
       "     'female',\n",
       "     'none',\n",
       "     3,\n",
       "     'real_estate',\n",
       "     29,\n",
       "     'none',\n",
       "     'own',\n",
       "     1,\n",
       "     'skilled',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     2.0,\n",
       "     4.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     [21,\n",
       "      [0, 1, 5, 7, 13, 14, 15, 16, 17, 18, 19, 20],\n",
       "      [2.0, 4.0, 1.0, 2.0, 14.0, 2368.0, 3.0, 3.0, 14.0, 29.0, 1.0, 1.0]],\n",
       "     [1.4143966876630347, -1.4143966876630347],\n",
       "     [0.9442120764123368, 0.05578792358766316],\n",
       "     0.0,\n",
       "     'No Risk'],\n",
       "    ['0_to_200',\n",
       "     4,\n",
       "     'no_credits',\n",
       "     'car_new',\n",
       "     250,\n",
       "     'less_100',\n",
       "     'unemployed',\n",
       "     2,\n",
       "     'female',\n",
       "     'none',\n",
       "     3,\n",
       "     'real_estate',\n",
       "     23,\n",
       "     'none',\n",
       "     'rent',\n",
       "     1,\n",
       "     'management_self-employed',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     2.0,\n",
       "     4.0,\n",
       "     4.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     [2.0,\n",
       "      4.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      4.0,\n",
       "      1.0,\n",
       "      0.0,\n",
       "      2.0,\n",
       "      0.0,\n",
       "      1.0,\n",
       "      2.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      4.0,\n",
       "      250.0,\n",
       "      2.0,\n",
       "      3.0,\n",
       "      4.0,\n",
       "      23.0,\n",
       "      1.0,\n",
       "      1.0],\n",
       "     [1.5173690203089494, -1.5173690203089494],\n",
       "     [0.9541190318258127, 0.0458809681741873],\n",
       "     0.0,\n",
       "     'No Risk'],\n",
       "    ['no_checking',\n",
       "     17,\n",
       "     'credits_paid_to_date',\n",
       "     'car_new',\n",
       "     832,\n",
       "     '100_to_500',\n",
       "     '1_to_4',\n",
       "     2,\n",
       "     'male',\n",
       "     'none',\n",
       "     2,\n",
       "     'real_estate',\n",
       "     42,\n",
       "     'none',\n",
       "     'own',\n",
       "     1,\n",
       "     'skilled',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     [21,\n",
       "      [1, 3, 7, 13, 14, 15, 16, 17, 18, 19, 20],\n",
       "      [1.0, 1.0, 2.0, 17.0, 832.0, 2.0, 2.0, 17.0, 42.0, 1.0, 1.0]],\n",
       "     [1.1122314956561121, -1.1122314956561121],\n",
       "     [0.9024248873010724, 0.0975751126989276],\n",
       "     0.0,\n",
       "     'No Risk'],\n",
       "    ['no_checking',\n",
       "     33,\n",
       "     'outstanding_credit',\n",
       "     'appliances',\n",
       "     5696,\n",
       "     'unknown',\n",
       "     'greater_7',\n",
       "     4,\n",
       "     'male',\n",
       "     'co-applicant',\n",
       "     4,\n",
       "     'unknown',\n",
       "     54,\n",
       "     'none',\n",
       "     'free',\n",
       "     2,\n",
       "     'skilled',\n",
       "     1,\n",
       "     'yes',\n",
       "     'yes',\n",
       "     0.0,\n",
       "     2.0,\n",
       "     2.0,\n",
       "     4.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     4.0,\n",
       "     1.0,\n",
       "     3.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     [0.0,\n",
       "      2.0,\n",
       "      4.0,\n",
       "      4.0,\n",
       "      2.0,\n",
       "      0.0,\n",
       "      1.0,\n",
       "      3.0,\n",
       "      0.0,\n",
       "      2.0,\n",
       "      0.0,\n",
       "      1.0,\n",
       "      0.0,\n",
       "      33.0,\n",
       "      5696.0,\n",
       "      4.0,\n",
       "      4.0,\n",
       "      33.0,\n",
       "      54.0,\n",
       "      2.0,\n",
       "      1.0],\n",
       "     [-1.3184238002172723, 1.3184238002172723],\n",
       "     [0.06680429265446867, 0.9331957073455314],\n",
       "     1.0,\n",
       "     'Risk'],\n",
       "    ['0_to_200',\n",
       "     13,\n",
       "     'prior_payments_delayed',\n",
       "     'retraining',\n",
       "     1375,\n",
       "     '100_to_500',\n",
       "     '4_to_7',\n",
       "     3,\n",
       "     'male',\n",
       "     'none',\n",
       "     3,\n",
       "     'real_estate',\n",
       "     37,\n",
       "     'none',\n",
       "     'own',\n",
       "     2,\n",
       "     'management_self-employed',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     2.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     8.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     [2.0,\n",
       "      0.0,\n",
       "      8.0,\n",
       "      1.0,\n",
       "      1.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      2.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      2.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      13.0,\n",
       "      1375.0,\n",
       "      3.0,\n",
       "      3.0,\n",
       "      13.0,\n",
       "      37.0,\n",
       "      2.0,\n",
       "      1.0],\n",
       "     [0.996270448775496, -0.996270448775496],\n",
       "     [0.8800116928980676, 0.11998830710193242],\n",
       "     0.0,\n",
       "     'No Risk']]}]}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy \n",
    "numpy.version.version\n",
    "\n",
    "import pandas as pd\n",
    "import json\n",
    "\n",
    "from pyspark import SparkContext, SQLContext\n",
    "from pyspark.ml import Pipeline\n",
    "from pyspark.ml.classification import RandomForestClassifier,GBTClassifier\n",
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "from pyspark.ml.feature import StringIndexer, VectorAssembler, IndexToString\n",
    "from pyspark.sql.types import StructType, DoubleType, StringType, ArrayType\n",
    "\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark import SparkFiles\n",
    "\n",
    "spark = SparkSession.builder.getOrCreate()\n",
    "pd_data = pd.read_csv(\"german_credit_data_biased_training.csv\", sep=\",\", header=0)\n",
    "spark_df = spark.read.csv(path=\"german_credit_data_biased_training.csv\", sep=\",\", header=True, inferSchema=True)\n",
    "spark_df.head()\n",
    "\n",
    "(train_data, test_data) = spark_df.randomSplit([0.9, 0.1], 24)\n",
    "print(\"Number of records for training: \" + str(train_data.count()))\n",
    "print(\"Number of records for evaluation: \" + str(test_data.count()))\n",
    "\n",
    "si_CheckingStatus = StringIndexer(inputCol='CheckingStatus', outputCol='CheckingStatus_IX')\n",
    "si_CreditHistory = StringIndexer(inputCol='CreditHistory', outputCol='CreditHistory_IX')\n",
    "si_LoanPurpose = StringIndexer(inputCol='LoanPurpose', outputCol='LoanPurpose_IX')\n",
    "si_ExistingSavings = StringIndexer(inputCol='ExistingSavings', outputCol='ExistingSavings_IX')\n",
    "si_EmploymentDuration = StringIndexer(inputCol='EmploymentDuration', outputCol='EmploymentDuration_IX')\n",
    "si_Sex = StringIndexer(inputCol='Sex', outputCol='Sex_IX')\n",
    "si_OthersOnLoan = StringIndexer(inputCol='OthersOnLoan', outputCol='OthersOnLoan_IX')\n",
    "si_OwnsProperty = StringIndexer(inputCol='OwnsProperty', outputCol='OwnsProperty_IX')\n",
    "si_InstallmentPlans = StringIndexer(inputCol='InstallmentPlans', outputCol='InstallmentPlans_IX')\n",
    "si_Housing = StringIndexer(inputCol='Housing', outputCol='Housing_IX')\n",
    "si_Job = StringIndexer(inputCol='Job', outputCol='Job_IX')\n",
    "si_Telephone = StringIndexer(inputCol='Telephone', outputCol='Telephone_IX')\n",
    "si_ForeignWorker = StringIndexer(inputCol='ForeignWorker', outputCol='ForeignWorker_IX')\n",
    "si_Label = StringIndexer(inputCol=\"Risk\", outputCol=\"label\").fit(spark_df)\n",
    "label_converter = IndexToString(inputCol=\"prediction\", outputCol=\"predictedLabel\", labels=si_Label.labels)\n",
    "\n",
    "va_features = VectorAssembler(\n",
    "inputCols=[\"CheckingStatus_IX\", \"CreditHistory_IX\", \"LoanPurpose_IX\", \"ExistingSavings_IX\",\n",
    "           \"EmploymentDuration_IX\", \"Sex_IX\", \"OthersOnLoan_IX\", \"OwnsProperty_IX\", \"InstallmentPlans_IX\",\n",
    "           \"Housing_IX\", \"Job_IX\", \"Telephone_IX\", \"ForeignWorker_IX\", \"LoanDuration\", \"LoanAmount\",\n",
    "           \"InstallmentPercent\", \"CurrentResidenceDuration\", \"LoanDuration\", \"Age\", \"ExistingCreditsCount\",\n",
    "           \"Dependents\"], outputCol=\"features\")\n",
    "\n",
    "classifier=GBTClassifier(featuresCol=\"features\")\n",
    "\n",
    "pipeline = Pipeline(\n",
    "stages=[si_CheckingStatus, si_CreditHistory, si_EmploymentDuration, si_ExistingSavings, si_ForeignWorker,\n",
    "        si_Housing, si_InstallmentPlans, si_Job, si_LoanPurpose, si_OthersOnLoan,\n",
    "        si_OwnsProperty, si_Sex, si_Telephone, si_Label, va_features, classifier, label_converter])\n",
    "\n",
    "model = pipeline.fit(train_data)\n",
    "predictions = model.transform(test_data)\n",
    "evaluator = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\")\n",
    "auc = evaluator.evaluate(predictions)\n",
    "\n",
    "print(\"Accuracy = %g\" % auc)\n",
    "\n",
    "# Remove existing model and deployment\n",
    "MODEL_NAME=PRE_PROD_MODEL_NAME\n",
    "DEPLOYMENT_NAME=PRE_PROD_DEPLOYMENT_NAME\n",
    "\n",
    "deployment_details = wml_client.deployments.get_details()\n",
    "for deployment in deployment_details['resources']:\n",
    "    deployment_id = wml_client.deployments.get_id(deployment)\n",
    "    model_id = deployment['entity']['asset']['id']\n",
    "    if deployment['entity']['name'] == DEPLOYMENT_NAME:\n",
    "        print('Deleting deployment id', deployment_id)\n",
    "        wml_client.deployments.delete(deployment_id)\n",
    "        print('Deleting model id', model_id)\n",
    "        wml_client.repository.delete(model_id)\n",
    "wml_client.repository.list_models()\n",
    "\n",
    "# Save Model\n",
    "software_spec_uid = wml_client.software_specifications.get_id_by_name(\"spark-mllib_3.3\")\n",
    "model_props_rf = {\n",
    "    wml_client.repository.ModelMetaNames.NAME: MODEL_NAME,\n",
    "    wml_client.repository.ModelMetaNames.DESCRIPTION: MODEL_NAME,\n",
    "    wml_client.repository.ModelMetaNames.SOFTWARE_SPEC_UID: software_spec_uid,\n",
    "    wml_client.repository.ModelMetaNames.TYPE: 'mllib_3.3'\n",
    "}\n",
    "print(model_props_rf)\n",
    "published_model_details = wml_client.repository.store_model(model=model, meta_props=model_props_rf, training_data=train_data, pipeline=pipeline)\n",
    "print(published_model_details)\n",
    "\n",
    "# List models in the repository\n",
    "wml_client.repository.list_models()\n",
    "\n",
    "# Get the model UID\n",
    "pre_prod_model_uid = wml_client.repository.get_model_id(published_model_details)\n",
    "pre_prod_model_uid\n",
    "\n",
    "\n",
    "# Deploy model\n",
    "wml_deployments = wml_client.deployments.get_details()\n",
    "pre_prod_deployment_uid = None\n",
    "for deployment in wml_deployments['resources']:\n",
    "    if DEPLOYMENT_NAME == deployment['entity']['name']:\n",
    "        pre_prod_deployment_uid = wml_client.deployments.get_id(deployment)\n",
    "        break\n",
    "\n",
    "if pre_prod_deployment_uid is None:\n",
    "    print(\"Deploying model...\")\n",
    "    meta_props = {\n",
    "        wml_client.deployments.ConfigurationMetaNames.NAME: DEPLOYMENT_NAME,\n",
    "        wml_client.deployments.ConfigurationMetaNames.DESCRIPTION: DEPLOYMENT_NAME,\n",
    "        wml_client.deployments.ConfigurationMetaNames.ONLINE: {}\n",
    "    }\n",
    "    deployment = wml_client.deployments.create(artifact_uid=pre_prod_model_uid, name=DEPLOYMENT_NAME, meta_props=meta_props)\n",
    "    pre_prod_deployment_uid = wml_client.deployments.get_id(deployment)\n",
    "\n",
    "print(\"Model id: {}\".format(pre_prod_model_uid))\n",
    "print(\"Deployment id: {}\".format(pre_prod_deployment_uid))\n",
    "\n",
    "pre_prod_deployment_uid=wml_client.deployments.get_id(deployment)\n",
    "pre_prod_deployment_uid\n",
    "\n",
    "fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\n",
    "values = [\n",
    "  [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n",
    "  [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n",
    "]\n",
    "\n",
    "payload_scoring = {\"fields\": fields,\"values\": values}\n",
    "payload = {\n",
    "    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n",
    "}\n",
    "scoring_response = wml_client.deployments.score(pre_prod_deployment_uid, payload)\n",
    "\n",
    "scoring_response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "60def73f-e179-4083-91c7-215debf39eba"
   },
   "source": [
    "## Deploy the Scikit-Learn Credit Risk Model to Watson Machine Learning\n",
    "\n",
    "The following cell deploys the Scikit-learn version of the German Credit Risk Model to the specified Machine Learning instance in the specified deployment space. This version of the German Credit Risk model has an auc-roc score around 85% and will be called the \"Challenger.\"\n",
    "\n",
    "Note : If you are running this notebook on python 3.10, then use `runtime-23.1-py3.10` and `scikit-learn_1.1` as software specifications name and ModelMetaNames.TYPE respectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "50193e3df87b49008440e9693efc2a47"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.1.0'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sklearn\n",
    "sklearn.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "178d5505-d7d6-44c7-83cd-3d72fe821adb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "     No Risk       0.92      0.80      0.86       382\n",
      "        Risk       0.55      0.78      0.65       118\n",
      "\n",
      "    accuracy                           0.80       500\n",
      "   macro avg       0.74      0.79      0.75       500\n",
      "weighted avg       0.83      0.80      0.81       500\n",
      "\n",
      "------------------------------------  -------------------------------------  ------------------------  ---------  ----------  ----------------\n",
      "ID                                    NAME                                   CREATED                   TYPE       SPEC_STATE  SPEC_REPLACEMENT\n",
      "67e11064-81b2-4ea8-addf-b39c218299b4  German Credit Risk Model - Challenger  2024-08-07T05:59:58.002Z  mllib_3.3  deprecated  spark-mllib_3.4\n",
      "f1b4fe35-1a75-4dca-a9b3-bc5dd7ed43db  German Credit Risk Model - PreProd     2024-08-07T05:59:14.002Z  mllib_3.3  deprecated  spark-mllib_3.4\n",
      "------------------------------------  -------------------------------------  ------------------------  ---------  ----------  ----------------\n",
      "{'entity': {'hybrid_pipeline_software_specs': [], 'label_column': 'Risk', 'schemas': {'input': [{'fields': [{'name': 'CheckingStatus', 'type': 'object'}, {'name': 'LoanDuration', 'type': 'int64'}, {'name': 'CreditHistory', 'type': 'object'}, {'name': 'LoanPurpose', 'type': 'object'}, {'name': 'LoanAmount', 'type': 'int64'}, {'name': 'ExistingSavings', 'type': 'object'}, {'name': 'EmploymentDuration', 'type': 'object'}, {'name': 'InstallmentPercent', 'type': 'int64'}, {'name': 'Sex', 'type': 'object'}, {'name': 'OthersOnLoan', 'type': 'object'}, {'name': 'CurrentResidenceDuration', 'type': 'int64'}, {'name': 'OwnsProperty', 'type': 'object'}, {'name': 'Age', 'type': 'int64'}, {'name': 'InstallmentPlans', 'type': 'object'}, {'name': 'Housing', 'type': 'object'}, {'name': 'ExistingCreditsCount', 'type': 'int64'}, {'name': 'Job', 'type': 'object'}, {'name': 'Dependents', 'type': 'int64'}, {'name': 'Telephone', 'type': 'object'}, {'name': 'ForeignWorker', 'type': 'object'}], 'id': '1', 'type': 'struct'}], 'output': []}, 'software_spec': {'id': '336b29df-e0e1-5e7d-b6a5-f6ab722625b2', 'name': 'runtime-23.1-py3.10'}, 'training_data_references': [{'id': '1', 'location': {}, 'schema': {'fields': [{'name': 'CheckingStatus', 'type': 'object'}, {'name': 'LoanDuration', 'type': 'int64'}, {'name': 'CreditHistory', 'type': 'object'}, {'name': 'LoanPurpose', 'type': 'object'}, {'name': 'LoanAmount', 'type': 'int64'}, {'name': 'ExistingSavings', 'type': 'object'}, {'name': 'EmploymentDuration', 'type': 'object'}, {'name': 'InstallmentPercent', 'type': 'int64'}, {'name': 'Sex', 'type': 'object'}, {'name': 'OthersOnLoan', 'type': 'object'}, {'name': 'CurrentResidenceDuration', 'type': 'int64'}, {'name': 'OwnsProperty', 'type': 'object'}, {'name': 'Age', 'type': 'int64'}, {'name': 'InstallmentPlans', 'type': 'object'}, {'name': 'Housing', 'type': 'object'}, {'name': 'ExistingCreditsCount', 'type': 'int64'}, {'name': 'Job', 'type': 'object'}, {'name': 'Dependents', 'type': 'int64'}, {'name': 'Telephone', 'type': 'object'}, {'name': 'ForeignWorker', 'type': 'object'}], 'id': '1', 'type': 'DataFrame'}, 'type': 'fs'}], 'type': 'scikit-learn_1.1'}, 'metadata': {'created_at': '2024-08-07T06:01:21.107Z', 'description': 'German Credit Risk Model - Challenger2', 'id': 'f1c77a66-34f2-4035-8733-d9795eb47912', 'modified_at': '2024-08-07T06:01:23.564Z', 'name': 'German Credit Risk Model - Challenger2', 'owner': '1000331001', 'resource_key': '42840a3c-e6da-4ae2-8809-6718f5638a9c', 'space_id': 'c3905865-4e73-43d1-a18f-24cf3a8b9163'}, 'system': {'warnings': []}}\n",
      "------------------------------------  --------------------------------------  ------------------------  ----------------  ----------  ----------------\n",
      "ID                                    NAME                                    CREATED                   TYPE              SPEC_STATE  SPEC_REPLACEMENT\n",
      "f1c77a66-34f2-4035-8733-d9795eb47912  German Credit Risk Model - Challenger2  2024-08-07T06:01:21.002Z  scikit-learn_1.1  supported\n",
      "67e11064-81b2-4ea8-addf-b39c218299b4  German Credit Risk Model - Challenger   2024-08-07T05:59:58.002Z  mllib_3.3         deprecated  spark-mllib_3.4\n",
      "f1b4fe35-1a75-4dca-a9b3-bc5dd7ed43db  German Credit Risk Model - PreProd      2024-08-07T05:59:14.002Z  mllib_3.3         deprecated  spark-mllib_3.4\n",
      "------------------------------------  --------------------------------------  ------------------------  ----------------  ----------  ----------------\n",
      "challenger_model_uid  f1c77a66-34f2-4035-8733-d9795eb47912\n",
      "Deploying model...\n",
      "\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "Synchronous deployment creation for uid: '67e11064-81b2-4ea8-addf-b39c218299b4' started\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "\n",
      "initializing\n",
      "Note: Software specification spark-mllib_3.3 is deprecated. Use spark-mllib_3.4 software specification instead when saving a spark model. For details, see https://www.ibm.com/support/producthub/icpdata/docs/content/SSQNUZ_latest/wsj/wmls/wmls-deploy-python-types.html.\n",
      "\n",
      "ready\n",
      "\n",
      "\n",
      "------------------------------------------------------------------------------------------------\n",
      "Successfully finished deployment creation, deployment_uid='f036196d-a13e-496c-9616-0cda347e2391'\n",
      "------------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "Model id: 67e11064-81b2-4ea8-addf-b39c218299b4\n",
      "Deployment id: None\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'predictions': [{'fields': ['CheckingStatus',\n",
       "    'LoanDuration',\n",
       "    'CreditHistory',\n",
       "    'LoanPurpose',\n",
       "    'LoanAmount',\n",
       "    'ExistingSavings',\n",
       "    'EmploymentDuration',\n",
       "    'InstallmentPercent',\n",
       "    'Sex',\n",
       "    'OthersOnLoan',\n",
       "    'CurrentResidenceDuration',\n",
       "    'OwnsProperty',\n",
       "    'Age',\n",
       "    'InstallmentPlans',\n",
       "    'Housing',\n",
       "    'ExistingCreditsCount',\n",
       "    'Job',\n",
       "    'Dependents',\n",
       "    'Telephone',\n",
       "    'ForeignWorker',\n",
       "    'CheckingStatus_IX',\n",
       "    'CreditHistory_IX',\n",
       "    'EmploymentDuration_IX',\n",
       "    'ExistingSavings_IX',\n",
       "    'ForeignWorker_IX',\n",
       "    'Housing_IX',\n",
       "    'InstallmentPlans_IX',\n",
       "    'Job_IX',\n",
       "    'LoanPurpose_IX',\n",
       "    'OthersOnLoan_IX',\n",
       "    'OwnsProperty_IX',\n",
       "    'Sex_IX',\n",
       "    'Telephone_IX',\n",
       "    'features',\n",
       "    'rawPrediction',\n",
       "    'probability',\n",
       "    'prediction',\n",
       "    'predictedLabel'],\n",
       "   'values': [['no_checking',\n",
       "     13,\n",
       "     'credits_paid_to_date',\n",
       "     'car_new',\n",
       "     1343,\n",
       "     '100_to_500',\n",
       "     '1_to_4',\n",
       "     2,\n",
       "     'female',\n",
       "     'none',\n",
       "     3,\n",
       "     'savings_insurance',\n",
       "     46,\n",
       "     'none',\n",
       "     'own',\n",
       "     2,\n",
       "     'skilled',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     [21,\n",
       "      [1, 3, 5, 13, 14, 15, 16, 17, 18, 19, 20],\n",
       "      [1.0, 1.0, 1.0, 13.0, 1343.0, 2.0, 3.0, 13.0, 46.0, 2.0, 1.0]],\n",
       "     [0.37923501094948475, -0.37923501094948475],\n",
       "     [0.6810214668555962, 0.31897853314440383],\n",
       "     0.0,\n",
       "     'No Risk'],\n",
       "    ['no_checking',\n",
       "     24,\n",
       "     'prior_payments_delayed',\n",
       "     'furniture',\n",
       "     4567,\n",
       "     '500_to_1000',\n",
       "     '1_to_4',\n",
       "     4,\n",
       "     'male',\n",
       "     'none',\n",
       "     4,\n",
       "     'savings_insurance',\n",
       "     36,\n",
       "     'none',\n",
       "     'free',\n",
       "     2,\n",
       "     'management_self-employed',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     [21,\n",
       "      [2, 3, 9, 10, 13, 14, 15, 16, 17, 18, 19, 20],\n",
       "      [1.0, 2.0, 2.0, 2.0, 24.0, 4567.0, 4.0, 4.0, 24.0, 36.0, 2.0, 1.0]],\n",
       "     [0.40678948974997703, -0.40678948974997703],\n",
       "     [0.6928716383154149, 0.3071283616845851],\n",
       "     0.0,\n",
       "     'No Risk'],\n",
       "    ['0_to_200',\n",
       "     26,\n",
       "     'all_credits_paid_back',\n",
       "     'car_new',\n",
       "     863,\n",
       "     'less_100',\n",
       "     'less_1',\n",
       "     2,\n",
       "     'female',\n",
       "     'co-applicant',\n",
       "     2,\n",
       "     'real_estate',\n",
       "     38,\n",
       "     'none',\n",
       "     'own',\n",
       "     1,\n",
       "     'skilled',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     2.0,\n",
       "     3.0,\n",
       "     3.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     2.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     [2.0,\n",
       "      3.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      3.0,\n",
       "      1.0,\n",
       "      1.0,\n",
       "      2.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      26.0,\n",
       "      863.0,\n",
       "      2.0,\n",
       "      2.0,\n",
       "      26.0,\n",
       "      38.0,\n",
       "      1.0,\n",
       "      1.0],\n",
       "     [1.08446701248213, -1.08446701248213],\n",
       "     [0.8974248814242887, 0.10257511857571133],\n",
       "     0.0,\n",
       "     'No Risk'],\n",
       "    ['0_to_200',\n",
       "     14,\n",
       "     'no_credits',\n",
       "     'car_new',\n",
       "     2368,\n",
       "     'less_100',\n",
       "     '1_to_4',\n",
       "     3,\n",
       "     'female',\n",
       "     'none',\n",
       "     3,\n",
       "     'real_estate',\n",
       "     29,\n",
       "     'none',\n",
       "     'own',\n",
       "     1,\n",
       "     'skilled',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     2.0,\n",
       "     4.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     [21,\n",
       "      [0, 1, 5, 7, 13, 14, 15, 16, 17, 18, 19, 20],\n",
       "      [2.0, 4.0, 1.0, 2.0, 14.0, 2368.0, 3.0, 3.0, 14.0, 29.0, 1.0, 1.0]],\n",
       "     [1.4143966876630347, -1.4143966876630347],\n",
       "     [0.9442120764123368, 0.05578792358766316],\n",
       "     0.0,\n",
       "     'No Risk'],\n",
       "    ['0_to_200',\n",
       "     4,\n",
       "     'no_credits',\n",
       "     'car_new',\n",
       "     250,\n",
       "     'less_100',\n",
       "     'unemployed',\n",
       "     2,\n",
       "     'female',\n",
       "     'none',\n",
       "     3,\n",
       "     'real_estate',\n",
       "     23,\n",
       "     'none',\n",
       "     'rent',\n",
       "     1,\n",
       "     'management_self-employed',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     2.0,\n",
       "     4.0,\n",
       "     4.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     [2.0,\n",
       "      4.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      4.0,\n",
       "      1.0,\n",
       "      0.0,\n",
       "      2.0,\n",
       "      0.0,\n",
       "      1.0,\n",
       "      2.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      4.0,\n",
       "      250.0,\n",
       "      2.0,\n",
       "      3.0,\n",
       "      4.0,\n",
       "      23.0,\n",
       "      1.0,\n",
       "      1.0],\n",
       "     [1.5173690203089494, -1.5173690203089494],\n",
       "     [0.9541190318258127, 0.0458809681741873],\n",
       "     0.0,\n",
       "     'No Risk'],\n",
       "    ['no_checking',\n",
       "     17,\n",
       "     'credits_paid_to_date',\n",
       "     'car_new',\n",
       "     832,\n",
       "     '100_to_500',\n",
       "     '1_to_4',\n",
       "     2,\n",
       "     'male',\n",
       "     'none',\n",
       "     2,\n",
       "     'real_estate',\n",
       "     42,\n",
       "     'none',\n",
       "     'own',\n",
       "     1,\n",
       "     'skilled',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     [21,\n",
       "      [1, 3, 7, 13, 14, 15, 16, 17, 18, 19, 20],\n",
       "      [1.0, 1.0, 2.0, 17.0, 832.0, 2.0, 2.0, 17.0, 42.0, 1.0, 1.0]],\n",
       "     [1.1122314956561121, -1.1122314956561121],\n",
       "     [0.9024248873010724, 0.0975751126989276],\n",
       "     0.0,\n",
       "     'No Risk'],\n",
       "    ['no_checking',\n",
       "     33,\n",
       "     'outstanding_credit',\n",
       "     'appliances',\n",
       "     5696,\n",
       "     'unknown',\n",
       "     'greater_7',\n",
       "     4,\n",
       "     'male',\n",
       "     'co-applicant',\n",
       "     4,\n",
       "     'unknown',\n",
       "     54,\n",
       "     'none',\n",
       "     'free',\n",
       "     2,\n",
       "     'skilled',\n",
       "     1,\n",
       "     'yes',\n",
       "     'yes',\n",
       "     0.0,\n",
       "     2.0,\n",
       "     2.0,\n",
       "     4.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     4.0,\n",
       "     1.0,\n",
       "     3.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     [0.0,\n",
       "      2.0,\n",
       "      4.0,\n",
       "      4.0,\n",
       "      2.0,\n",
       "      0.0,\n",
       "      1.0,\n",
       "      3.0,\n",
       "      0.0,\n",
       "      2.0,\n",
       "      0.0,\n",
       "      1.0,\n",
       "      0.0,\n",
       "      33.0,\n",
       "      5696.0,\n",
       "      4.0,\n",
       "      4.0,\n",
       "      33.0,\n",
       "      54.0,\n",
       "      2.0,\n",
       "      1.0],\n",
       "     [-1.3184238002172723, 1.3184238002172723],\n",
       "     [0.06680429265446867, 0.9331957073455314],\n",
       "     1.0,\n",
       "     'Risk'],\n",
       "    ['0_to_200',\n",
       "     13,\n",
       "     'prior_payments_delayed',\n",
       "     'retraining',\n",
       "     1375,\n",
       "     '100_to_500',\n",
       "     '4_to_7',\n",
       "     3,\n",
       "     'male',\n",
       "     'none',\n",
       "     3,\n",
       "     'real_estate',\n",
       "     37,\n",
       "     'none',\n",
       "     'own',\n",
       "     2,\n",
       "     'management_self-employed',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     2.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     8.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     [2.0,\n",
       "      0.0,\n",
       "      8.0,\n",
       "      1.0,\n",
       "      1.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      2.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      2.0,\n",
       "      0.0,\n",
       "      0.0,\n",
       "      13.0,\n",
       "      1375.0,\n",
       "      3.0,\n",
       "      3.0,\n",
       "      13.0,\n",
       "      37.0,\n",
       "      2.0,\n",
       "      1.0],\n",
       "     [0.996270448775496, -0.996270448775496],\n",
       "     [0.8800116928980676, 0.11998830710193242],\n",
       "     0.0,\n",
       "     'No Risk']]}]}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import json\n",
    "import sys\n",
    "import numpy\n",
    "import sklearn\n",
    "import sklearn.ensemble\n",
    "numpy.set_printoptions(threshold=sys.maxsize)\n",
    "#from sklearn.ensemble.gradient_boosting import GradientBoostingClassifier\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.utils.multiclass import type_of_target\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import StandardScaler, OrdinalEncoder\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.model_selection import cross_validate\n",
    "from sklearn.metrics import get_scorer\n",
    "from sklearn.model_selection import cross_validate\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "data_df=pd.read_csv (\"german_credit_data_biased_training.csv\")\n",
    "\n",
    "data_df.head()\n",
    "\n",
    "target_label_name = \"Risk\"\n",
    "feature_cols= data_df.drop(columns=[target_label_name])\n",
    "label= data_df[target_label_name]\n",
    "\n",
    "# Set model evaluation properties\n",
    "optimization_metric = 'roc_auc'\n",
    "random_state = 33\n",
    "cv_num_folds = 3\n",
    "holdout_fraction = 0.1\n",
    "\n",
    "if type_of_target(label.values) in ['multiclass', 'binary']:\n",
    "    X_train, X_holdout, y_train, y_holdout = train_test_split(feature_cols, label, test_size=holdout_fraction, random_state=random_state, stratify=label.values)\n",
    "else:\n",
    "    X_train, X_holdout, y_train, y_holdout = train_test_split(feature_cols, label, test_size=holdout_fraction, random_state=random_state)\n",
    "\n",
    "# Data preprocessing transformer generation\n",
    "\n",
    "numeric_transformer = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='median')),\n",
    "    ('scaler', StandardScaler())])\n",
    "categorical_transformer = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='most_frequent')),\n",
    "    ('OrdinalEncoder', OrdinalEncoder(categories='auto',dtype=numpy.float64 ))])\n",
    "\n",
    "numeric_features = feature_cols.select_dtypes(include=['int64', 'float64']).columns\n",
    "categorical_features = feature_cols.select_dtypes(include=['object']).columns\n",
    "\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('num', numeric_transformer, numeric_features),\n",
    "        ('cat', categorical_transformer, categorical_features)])\n",
    "\n",
    "# Initiate model and create pipeline\n",
    "model=GradientBoostingClassifier()\n",
    "gbt_pipeline = Pipeline(steps=[('preprocessor', preprocessor), ('classifier', model)])\n",
    "model_gbt=gbt_pipeline.fit(X_train, y_train)\n",
    "\n",
    "y_pred = model_gbt.predict(X_holdout)\n",
    "\n",
    "\n",
    "# Evaluate model performance on test data and Cross validation\n",
    "scorer = get_scorer(optimization_metric)\n",
    "scorer(model_gbt,X_holdout, y_holdout)\n",
    "\n",
    "# Cross validation -3 folds\n",
    "cv_results = cross_validate(model_gbt,X_train,y_train, scoring={optimization_metric:scorer})\n",
    "numpy.mean(cv_results['test_' + optimization_metric])\n",
    "\n",
    "print(classification_report(y_pred, y_holdout))\n",
    "\n",
    "\n",
    "# Remove existing model and deployment\n",
    "MODEL_NAME=PRE_PROD_CHALLENGER_MODEL_NAME\n",
    "DEPLOYMENT_NAME=PRE_PROD_CHALLENGER_DEPLOYMENT_NAME\n",
    "\n",
    "deployment_details = wml_client.deployments.get_details()\n",
    "for deployment in deployment_details['resources']:\n",
    "    deployment_id = wml_client.deployments.get_id(deployment)\n",
    "    model_id = deployment['entity']['asset']['id']\n",
    "    if deployment['entity']['name'] == DEPLOYMENT_NAME:\n",
    "        print('Deleting deployment id', deployment_id)\n",
    "        wml_client.deployments.delete(deployment_id)\n",
    "        print('Deleting model id', model_id)\n",
    "        wml_client.repository.delete(model_id)\n",
    "wml_client.repository.list_models()\n",
    "\n",
    "# Store Model\n",
    "#Note if there is specification related exception then use \"default_py3.7_opence\" instead of default_py3.8\n",
    "software_spec_uid = wml_client.software_specifications.get_id_by_name(\"runtime-23.1-py3.10\")\n",
    "model_props_gbt = {\n",
    "    wml_client.repository.ModelMetaNames.NAME: MODEL_NAME,\n",
    "    wml_client.repository.ModelMetaNames.DESCRIPTION: MODEL_NAME,\n",
    "    wml_client.repository.ModelMetaNames.SOFTWARE_SPEC_UID: software_spec_uid,\n",
    "    wml_client.repository.ModelMetaNames.TYPE: \"scikit-learn_1.1\"\n",
    "}\n",
    "\n",
    "published_model_details = wml_client.repository.store_model(model=model_gbt, meta_props=model_props_gbt, training_data=feature_cols,training_target=label)\n",
    "print(published_model_details)\n",
    "\n",
    "# List models in the repository\n",
    "wml_client.repository.list_models()\n",
    "\n",
    "# Get the model UID\n",
    "challenger_model_uid = wml_client.repository.get_model_id(published_model_details)\n",
    "print('challenger_model_uid ', challenger_model_uid)\n",
    "\n",
    "\n",
    "# Deploy model\n",
    "wml_deployments = wml_client.deployments.get_details()\n",
    "challenger_deployment_uid = None\n",
    "for deployment in wml_deployments['resources']:\n",
    "    if DEPLOYMENT_NAME == deployment['entity']['name']:\n",
    "        challenger_deployment_uid = wml_client.deployments.get_id(deployment)\n",
    "        break\n",
    "\n",
    "if challenger_deployment_uid is None:\n",
    "    print(\"Deploying model...\")\n",
    "    meta_props = {\n",
    "        wml_client.deployments.ConfigurationMetaNames.NAME: DEPLOYMENT_NAME,\n",
    "        wml_client.deployments.ConfigurationMetaNames.DESCRIPTION: DEPLOYMENT_NAME,\n",
    "        wml_client.deployments.ConfigurationMetaNames.ONLINE: {}\n",
    "    }\n",
    "    deployment = wml_client.deployments.create(artifact_uid=challenger_model_uid, meta_props=meta_props)\n",
    "    challenger_deployment_uid = wml_client.deployments.get_id(deployment)\n",
    "\n",
    "print(\"Model id: {}\".format(challenger_model_uid))\n",
    "print(\"Deployment id: {}\".format(challenger_deployment_uid))\n",
    "\n",
    "challenger_deployment_uid=wml_client.deployments.get_id(deployment)\n",
    "challenger_deployment_uid\n",
    "\n",
    "\n",
    "# Sample scoring\n",
    "fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\n",
    "values = [\n",
    "  [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n",
    "  [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n",
    "]\n",
    "\n",
    "payload_scoring = {\"fields\": fields,\"values\": values}\n",
    "payload = {\n",
    "    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n",
    "}\n",
    "scoring_response = wml_client.deployments.score(challenger_deployment_uid, payload)\n",
    "\n",
    "scoring_response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6d66e647-0b3e-483e-8223-55d79934f20c"
   },
   "source": [
    "# Configure OpenScale \n",
    "The notebook will now import the necessary libraries and set up a Python OpenScale client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "94363246-f6e0-4a95-af54-9d8513854dbe"
   },
   "outputs": [],
   "source": [
    "from ibm_cloud_sdk_core.authenticators import CloudPakForDataAuthenticator\n",
    "from ibm_watson_openscale import APIClient\n",
    "from ibm_watson_openscale.utils import *\n",
    "from ibm_watson_openscale.supporting_classes import *\n",
    "from ibm_watson_openscale.supporting_classes.enums import *\n",
    "from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord\n",
    "import uuid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "9c822f42-7763-4fe5-bbf7-ed6fb6ae4722"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3.0.40'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "authenticator = CloudPakForDataAuthenticator(\n",
    "        url=WOS_CREDENTIALS['url'],\n",
    "        username=WOS_CREDENTIALS['username'],\n",
    "        password=WOS_CREDENTIALS['password'],\n",
    "        disable_ssl_verification=True\n",
    "    )\n",
    "\n",
    "wos_client = APIClient(service_url=WOS_CREDENTIALS['url'],authenticator=authenticator)\n",
    "wos_client.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "444021dc-f467-4b55-9c19-2a42e2725985"
   },
   "source": [
    "## Create schema and datamart\n",
    "\n",
    "### Set up datamart\n",
    "Watson OpenScale uses a database to store payload logs and calculated metrics. If an OpenScale datamart exists in Db2, the existing datamart will be used and no data will be overwritten."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "78d73bc1b8f84765bf0eb36e4c637d17"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Data Marts</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>description</th><th style='border: 1px solid #dddddd'>internal_database</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>AIOSFASTPATHICP-00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>Data Mart created by OpenScale ExpressPath</td><td style='border: 1px solid #dddddd'>False</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-06-04 05:19:03.698000+00:00</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wos_client.data_marts.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "af7f338d-cd56-45f9-8817-40bb176d523d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using existing datamart 00000000-0000-0000-0000-000000000000\n"
     ]
    }
   ],
   "source": [
    "data_marts = wos_client.data_marts.list().result.data_marts\n",
    "if len(data_marts) == 0:\n",
    "    if DATABASE_CREDENTIALS is not None:\n",
    "        if SCHEMA_NAME is None: \n",
    "            print(\"Please specify the SCHEMA_NAME and rerun the cell\")\n",
    "\n",
    "        print(\"Setting up external datamart\")\n",
    "        added_data_mart_result = wos_client.data_marts.add(\n",
    "                background_mode=False,\n",
    "                name=\"WOS Data Mart\",\n",
    "                description=\"Data Mart created by WOS tutorial notebook\",\n",
    "                database_configuration=DatabaseConfigurationRequest(\n",
    "                  database_type=DatabaseType.DB2,\n",
    "                    credentials=PrimaryStorageCredentialsLong(\n",
    "                        hostname=DATABASE_CREDENTIALS[\"hostname\"],\n",
    "                        username=DATABASE_CREDENTIALS[\"username\"],\n",
    "                        password=DATABASE_CREDENTIALS[\"password\"],\n",
    "                        db=DATABASE_CREDENTIALS[\"db\"],\n",
    "                        port=DATABASE_CREDENTIALS[\"port\"]\n",
    "                    ),\n",
    "                    location=LocationSchemaName(\n",
    "                        schema_name= SCHEMA_NAME\n",
    "                    )\n",
    "                )\n",
    "             ).result\n",
    "    else:\n",
    "        print(\"Setting up internal datamart\")\n",
    "        added_data_mart_result = wos_client.data_marts.add(\n",
    "                background_mode=False,\n",
    "                name=\"WOS Data Mart\",\n",
    "                description=\"Data Mart created by WOS tutorial notebook\", \n",
    "                internal_database = True).result\n",
    "        \n",
    "    data_mart_id = added_data_mart_result.metadata.id\n",
    "    \n",
    "else:\n",
    "    data_mart_id=data_marts[0].metadata.id\n",
    "    print(\"Using existing datamart {}\".format(data_mart_id))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "4a6c18b3207f4ff98bf21621df5a2ec9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'00000000-0000-0000-0000-000000000000'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_mart_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "59cf3a5b-78be-4d10-8ae8-069a98b48fa2"
   },
   "source": [
    "## Generate an ICP token\n",
    "\n",
    "The following is a function that will generate an ICP access token used to interact with the Watson OpenScale APIs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "9e6c7b03-7506-4ad4-b656-7f29e2349ea8"
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import urllib3\n",
    "from http import HTTPStatus\n",
    "\n",
    "def get_iamtoken(url, username, password):\n",
    "    fqdn = urllib3.util.parse_url(url).netloc\n",
    "    domain = '.'.join(fqdn.split('.')[1:])\n",
    "    token_url = 'https://cp-console.{}/idprovider/v1/auth/identitytoken'.format(domain)\n",
    "    data = {\n",
    "        'grant_type': 'password',\n",
    "        'username': username,\n",
    "        'password': password,\n",
    "        'scope': 'openid'\n",
    "    }\n",
    "    return requests.post(token_url, data, verify=False)\n",
    "\n",
    "def get_accesstoken(url, username, iamtoken):\n",
    "    url = '{}/v1/preauth/validateAuth'.format(url)\n",
    "    headers = {\n",
    "        'Content-type': 'application/json',\n",
    "        'username': username,\n",
    "        'iam-token': iamtoken\n",
    "    }\n",
    "    return requests.get(url, headers=headers, verify=False)\n",
    "\n",
    "def generate_access_token():\n",
    "    url=WOS_CREDENTIALS['url']\n",
    "    username=WOS_CREDENTIALS['username']\n",
    "    password=WOS_CREDENTIALS['password'] \n",
    "    response = get_iamtoken(url,username,password)\n",
    "    #service is not available when iamintegration=false so fall back to old way of generating code\n",
    "    if response.status_code==HTTPStatus.SERVICE_UNAVAILABLE:\n",
    "        url = '{}/v1/preauth/validateAuth'.format(url)\n",
    "        headers = {'Content-type': 'application/json'}\n",
    "        data = {\n",
    "            'grant_type': 'password',\n",
    "            'username': username,\n",
    "            'password': password\n",
    "        }\n",
    "        return requests.get(url, headers=headers, auth= (username,password),verify=False).json()['accessToken']\n",
    "        \n",
    "    else:\n",
    "        return get_accesstoken(url,username, response.json()['access_token']).json()['accessToken']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4b0c9239-899e-4748-80e2-b901a4d26741"
   },
   "source": [
    "## Bind WML machine learning instance as Pre-Prod\n",
    "\n",
    "Watson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model. If a binding with name \"WML Pre-Prod\" already exists, this code will delete that binding a create a new one.\n",
    "\n",
    "**Note**: Binding with name `WML Pre-Prod` is assumed to be only created by this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "d97136a95b184de8b794d14318367aad"
   },
   "outputs": [],
   "source": [
    "SERVICE_PROVIDER_NAME = \"WML Pre-Prod\"\n",
    "SERVICE_PROVIDER_DESCRIPTION = \"Added by tutorial WOS notebook.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "33c581b9-fc49-4885-90d9-d25e2f78762a"
   },
   "outputs": [],
   "source": [
    "service_providers = wos_client.service_providers.list().result.service_providers\n",
    "for service_provider in service_providers:\n",
    "    service_instance_name = service_provider.entity.name\n",
    "    if service_instance_name == SERVICE_PROVIDER_NAME:\n",
    "        service_provider_id = service_provider.metadata.id\n",
    "        wos_client.service_providers.delete(service_provider_id)\n",
    "        print(\"Deleted existing service_provider for WML instance: {}\".format(service_provider_id))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "60add123-1273-4190-97bd-1c3ff5769fba"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "=================================================================================\n",
      "\n",
      " Waiting for end of adding service provider 81655d73-6ea3-44ae-9876-86df1fa6390d \n",
      "\n",
      "=================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "-----------------------------------------------\n",
      " Successfully finished adding service provider \n",
      "-----------------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "added_service_provider_result = wos_client.service_providers.add(\n",
    "        name=SERVICE_PROVIDER_NAME,\n",
    "        description=SERVICE_PROVIDER_DESCRIPTION,\n",
    "        service_type=ServiceTypes.WATSON_MACHINE_LEARNING,\n",
    "        deployment_space_id = space_id,\n",
    "        operational_space_id = \"pre_production\",\n",
    "        credentials=WMLCredentialsCP4D(),\n",
    "        background_mode=False\n",
    "    ).result\n",
    "service_provider_id = added_service_provider_result.metadata.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "3bb0b46e-a5d3-4fff-a746-7d3aae5a7d1f"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Service Providers</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>instance_id</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>service_type</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>99999999-9999-9999-9999-999999999999</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>WML Pre-Prod2</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-08-07 06:03:11.179000+00:00</td><td style='border: 1px solid #dddddd'>81655d73-6ea3-44ae-9876-86df1fa6390d</td></tr><tr><td style='border: 1px solid #dddddd'>99999999-9999-9999-9999-999999999999</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>WML AI function -  WOS notebook</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-08-06 13:32:38.884000+00:00</td><td style='border: 1px solid #dddddd'>e413a76d-802c-44f3-8c55-300c5aa2659b</td></tr><tr><td style='border: 1px solid #dddddd'>99999999-9999-9999-9999-999999999999</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>Watson Machine Learning - Indirect Bias Demo-2</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-08-06 13:25:37.046000+00:00</td><td style='border: 1px solid #dddddd'>c3601bc6-22c9-45a1-b016-c29cf0bf64b4</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>IAE7</td><td style='border: 1px solid #dddddd'>custom_machine_learning</td><td style='border: 1px solid #dddddd'>2024-07-02 09:08:06.273000+00:00</td><td style='border: 1px solid #dddddd'>184e73a2-7fd8-4f3f-b994-bb648f6eb8ec</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>IAE6</td><td style='border: 1px solid #dddddd'>custom_machine_learning</td><td style='border: 1px solid #dddddd'>2024-07-02 09:04:22.643000+00:00</td><td style='border: 1px solid #dddddd'>644befcd-6d36-4f4d-a30a-cd51a28b63fe</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>WML_IAE5</td><td style='border: 1px solid #dddddd'>custom_machine_learning</td><td style='border: 1px solid #dddddd'>2024-07-02 08:43:08.723000+00:00</td><td style='border: 1px solid #dddddd'>e986a0d7-8187-4fed-ab2a-c614a9683cae</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>WML_IAE4</td><td style='border: 1px solid #dddddd'>custom_machine_learning</td><td style='border: 1px solid #dddddd'>2024-07-02 07:00:41.816000+00:00</td><td style='border: 1px solid #dddddd'>d90c6bf2-49c6-4179-9876-8b85b0247d95</td></tr><tr><td style='border: 1px solid #dddddd'>99999999-9999-9999-9999-999999999999</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>Image Multiclass Watson Machine Learning V2_test</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-07-01 17:17:14.696000+00:00</td><td style='border: 1px solid #dddddd'>a7ca157a-de07-457a-8c4c-b1a2e998699c</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>WML_remote_spark_jdbc</td><td style='border: 1px solid #dddddd'>custom_machine_learning</td><td style='border: 1px solid #dddddd'>2024-07-01 09:40:35.190000+00:00</td><td style='border: 1px solid #dddddd'>26b2af15-e396-4295-81d2-6912ab93912b</td></tr><tr><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>WML_IAE3</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-06-30 11:22:00.507000+00:00</td><td style='border: 1px solid #dddddd'>d7ad1164-a387-462d-b495-dd25d6241404</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: First 10 records were displayed.\n"
     ]
    }
   ],
   "source": [
    "wos_client.service_providers.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7ebeaba2-1ef8-441b-8a70-cb8bf8c73865"
   },
   "source": [
    "## Subscriptions\n",
    "### Remove existing PreProd and Challenger credit risk subscriptions\n",
    "This code removes previous subscriptions with name `German Credit Risk Model - PreProd` and `German Credit Risk Model - Challenger` to refresh the monitors with the new model and new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "987e9389-4cb2-4817-97e6-d50fe7a241ed"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleted existing subscription for German Credit Risk Model - Challenger\n",
      "Deleted existing subscription for German Credit Risk Model - PreProd\n"
     ]
    }
   ],
   "source": [
    "subscriptions = wos_client.subscriptions.list().result.subscriptions\n",
    "for subscription in subscriptions:\n",
    "    sub_name = subscription.entity.asset.name\n",
    "    if sub_name == PRE_PROD_MODEL_NAME or sub_name == PRE_PROD_CHALLENGER_MODEL_NAME:\n",
    "        wos_client.subscriptions.delete(subscription.metadata.id)\n",
    "        print('Deleted existing subscription for', sub_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "id": "32b7805b67ea46af83e2e0cb55b3f677"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Subscriptions</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>asset_id</th><th style='border: 1px solid #dddddd'>asset_type</th><th style='border: 1px solid #dddddd'>asset_name</th><th style='border: 1px solid #dddddd'>data_mart_id</th><th style='border: 1px solid #dddddd'>deployment_id</th><th style='border: 1px solid #dddddd'>deployment_name</th><th style='border: 1px solid #dddddd'>service_provider_id</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>a4a1a71d-2a44-43f8-b92a-ba3c80dc36fd</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Credit Risk python Fn Model</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>90b19695-de03-41b9-b820-2d3e11c2f662</td><td style='border: 1px solid #dddddd'>dep_Credit Risk python Fn Model</td><td style='border: 1px solid #dddddd'>e413a76d-802c-44f3-8c55-300c5aa2659b</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-08-06 13:32:53.985000+00:00</td><td style='border: 1px solid #dddddd'>77c99979-8408-40b7-ab3a-e3c0a8a80aab</td></tr><tr><td style='border: 1px solid #dddddd'>eed8b225-19e5-4460-9b6f-7271dc1e3ff2</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Adult Census Income Classifier Model</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>038b8c68-6615-4c0d-8d46-15783d6ab9c8</td><td style='border: 1px solid #dddddd'>Adult Census Income Classifier Deployment</td><td style='border: 1px solid #dddddd'>c3601bc6-22c9-45a1-b016-c29cf0bf64b4</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-08-06 13:26:09.549000+00:00</td><td style='border: 1px solid #dddddd'>c274e76c-6bb9-43ce-ba64-127a3db95ede</td></tr><tr><td style='border: 1px solid #dddddd'>438ca544-9bd1-48c2-8e8d-3de4ef4ca79b</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>WML_IAE4</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>78a0af9e-1014-4fb1-b22a-5e11f4fd70e7</td><td style='border: 1px solid #dddddd'>WML_IAE4</td><td style='border: 1px solid #dddddd'>d90c6bf2-49c6-4179-9876-8b85b0247d95</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-02 07:03:31.504000+00:00</td><td style='border: 1px solid #dddddd'>e34b9b87-b6e1-4c53-b92e-cb80dea042be</td></tr><tr><td style='border: 1px solid #dddddd'>592b902d-3dc9-4e56-8bcb-86cbf1a6d8a9</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>gcr - P2 XGB Classifier - Model</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>2b976af0-e4ab-4859-af7d-2f2287d864ad</td><td style='border: 1px solid #dddddd'>gcr model</td><td style='border: 1px solid #dddddd'>4d2f2fb2-6b64-4d58-8f13-257166e468e9</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-17 07:11:44.727000+00:00</td><td style='border: 1px solid #dddddd'>e2df4ec7-6c75-416f-a444-8d21389f7513</td></tr><tr><td style='border: 1px solid #dddddd'>e3ac9fc3-bccf-4a4e-b37b-490bfb93dd81</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>GCR AutoAI - P2 XGB Classifier - Model</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>6399e6e8-df5a-4370-9af4-34b2f2e76bc6</td><td style='border: 1px solid #dddddd'>GCR Auto AI</td><td style='border: 1px solid #dddddd'>a7ca157a-de07-457a-8c4c-b1a2e998699c</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-03 10:37:20.487000+00:00</td><td style='border: 1px solid #dddddd'>ce36911c-75f6-4f99-b4d2-b71e5d55a802</td></tr><tr><td style='border: 1px solid #dddddd'>592b902d-3dc9-4e56-8bcb-86cbf1a6d8a9</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>gcr - P2 XGB Classifier - Model</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>2b976af0-e4ab-4859-af7d-2f2287d864ad</td><td style='border: 1px solid #dddddd'>gcr model</td><td style='border: 1px solid #dddddd'>4d2f2fb2-6b64-4d58-8f13-257166e468e9</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-02 15:33:17.454000+00:00</td><td style='border: 1px solid #dddddd'>e96278a6-7190-48f3-b8bd-945fa48cfe50</td></tr><tr><td style='border: 1px solid #dddddd'>327d8aea-ecfc-4990-9bb9-601a1695094d</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>GCR AutoAI - P2 XGB Classifier - Model</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>755c3e75-24b5-4839-8a8f-3f85c07a40c9</td><td style='border: 1px solid #dddddd'>GCR demo</td><td style='border: 1px solid #dddddd'>a7ca157a-de07-457a-8c4c-b1a2e998699c</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-02 12:03:14.224000+00:00</td><td style='border: 1px solid #dddddd'>a0f86241-8bfc-4322-895a-2597512e1653</td></tr><tr><td style='border: 1px solid #dddddd'>b21904ef-7478-4dae-b93f-c120e95c9200</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>My SDK Batch Subscription-db2</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>a10a121b-2394-4fe7-9a2d-f0520abd212c</td><td style='border: 1px solid #dddddd'>My SDK Batch Subscription-db2</td><td style='border: 1px solid #dddddd'>644befcd-6d36-4f4d-a30a-cd51a28b63fe</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-02 09:04:38.883000+00:00</td><td style='border: 1px solid #dddddd'>70c9c394-2530-4c0f-b0b2-6b81e44612d0</td></tr><tr><td style='border: 1px solid #dddddd'>7bf1d492-3275-405d-9b61-da4e2075e746</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>My SDK Batch Subscription-db2</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>a302d0d4-3f38-4ded-8688-3e4b2e9bb55c</td><td style='border: 1px solid #dddddd'>My SDK Batch Subscription-db2</td><td style='border: 1px solid #dddddd'>e986a0d7-8187-4fed-ab2a-c614a9683cae</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-02 08:43:25.526000+00:00</td><td style='border: 1px solid #dddddd'>f4178567-4f9a-4b0e-b255-2babdb8b5f38</td></tr><tr><td style='border: 1px solid #dddddd'>405fe789-e294-42ce-9989-774d484205c3</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>WML_remote_spark_jdbc</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>ea8d064d-2eb9-48a0-97af-f385dd358843</td><td style='border: 1px solid #dddddd'>WML_remote_spark_jdbc</td><td style='border: 1px solid #dddddd'>26b2af15-e396-4295-81d2-6912ab93912b</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-01 09:41:24.463000+00:00</td><td style='border: 1px solid #dddddd'>3046b546-e422-4737-9123-a26e025d3cef</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: First 10 records were displayed.\n"
     ]
    }
   ],
   "source": [
    "wos_client.subscriptions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "3b900ee8ed16456e82db4c5d45ef88fe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: Software specification spark-mllib_3.3 is deprecated. Use spark-mllib_3.4 software specification instead when saving a spark model. For details, see https://www.ibm.com/support/producthub/icpdata/docs/content/SSQNUZ_latest/wsj/wmls/wmls-deploy-python-types.html.\n"
     ]
    }
   ],
   "source": [
    "# Create preprod subscription\n",
    "pre_prod_deployment = wml_client.deployments.get_details(deployment_uid=pre_prod_deployment_uid)\n",
    "pre_prod_asset_details=wos_client.service_providers.get_deployment_asset(data_mart_id=data_mart_id,service_provider_id=service_provider_id,deployment_id=pre_prod_deployment_uid,deployment_space_id=space_id)\n",
    "pre_prod_asset = Asset(\n",
    "    asset_id=pre_prod_asset_details[\"entity\"][\"asset\"][\"asset_id\"],\n",
    "    name=pre_prod_asset_details[\"entity\"][\"asset\"][\"name\"],\n",
    "    url=pre_prod_asset_details[\"entity\"][\"asset\"][\"url\"],\n",
    "    asset_type=AssetTypes.MODEL,\n",
    "    input_data_type=InputDataType.STRUCTURED,\n",
    "    problem_type=ProblemType.BINARY_CLASSIFICATION,\n",
    "    model_type='mllib_3.3'\n",
    ")\n",
    "pre_prod_asset_deployment = AssetDeploymentRequest(\n",
    "    deployment_id=pre_prod_deployment_uid,\n",
    "    name=PRE_PROD_DEPLOYMENT_NAME,\n",
    "    deployment_type=DeploymentTypes.ONLINE,\n",
    "    url=pre_prod_deployment[\"entity\"][\"status\"][\"online_url\"][\"url\"]\n",
    ")\n",
    "\n",
    "pre_prod_asset_properties_request = AssetPropertiesRequest(\n",
    "    label_column=\"Risk\",\n",
    "    probability_fields=[\"probability\"],\n",
    "    prediction_field=\"predictedLabel\",\n",
    "    feature_fields=[\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"],\n",
    "    categorical_fields=[\"CheckingStatus\",\"CreditHistory\",\"LoanPurpose\",\"ExistingSavings\",\"EmploymentDuration\",\"Sex\",\"OthersOnLoan\",\"OwnsProperty\",\"InstallmentPlans\",\"Housing\",\"Job\",\"Telephone\",\"ForeignWorker\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "id": "d3902de0ac9c4707b975623eac635ac3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'metadata': {'guid': '42b13344-baa1-45a0-89ca-ef662387f691',\n",
       "  'created_at': '2024-08-07T05:59:28.691Z',\n",
       "  'modified_at': '2024-08-07T05:59:28.691Z'},\n",
       " 'entity': {'name': 'German Credit Risk Model - PreProd',\n",
       "  'type': 'online',\n",
       "  'description': 'German Credit Risk Model - PreProd',\n",
       "  'scoring_endpoint': {'url': 'https://internal-nginx-svc:12443/ml/v4/deployments/42b13344-baa1-45a0-89ca-ef662387f691/predictions'},\n",
       "  'asset': {'asset_id': 'f1b4fe35-1a75-4dca-a9b3-bc5dd7ed43db',\n",
       "   'url': 'https://internal-nginx-svc:12443/ml/v4/models/f1b4fe35-1a75-4dca-a9b3-bc5dd7ed43db?space_id=c3905865-4e73-43d1-a18f-24cf3a8b9163&version=2020-06-12',\n",
       "   'name': 'German Credit Risk Model - PreProd',\n",
       "   'asset_type': 'model',\n",
       "   'created_at': '2024-08-07T05:59:14.241Z',\n",
       "   'modified_at': '2024-08-07T05:59:22.805Z'},\n",
       "  'asset_properties': {'model_type': 'mllib_3.3',\n",
       "   'runtime_environment': 'spark-3.3.0',\n",
       "   'label_column': 'Risk',\n",
       "   'input_data_schema': {'type': 'struct',\n",
       "    'id': '1',\n",
       "    'fields': [{'name': 'CheckingStatus',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'LoanDuration',\n",
       "      'type': 'integer',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'CreditHistory',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'LoanPurpose',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'LoanAmount',\n",
       "      'type': 'integer',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'ExistingSavings',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'EmploymentDuration',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'InstallmentPercent',\n",
       "      'type': 'integer',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'Sex', 'type': 'string', 'nullable': True, 'metadata': {}},\n",
       "     {'name': 'OthersOnLoan',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'CurrentResidenceDuration',\n",
       "      'type': 'integer',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'OwnsProperty',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'Age', 'type': 'integer', 'nullable': True, 'metadata': {}},\n",
       "     {'name': 'InstallmentPlans',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'Housing', 'type': 'string', 'nullable': True, 'metadata': {}},\n",
       "     {'name': 'ExistingCreditsCount',\n",
       "      'type': 'integer',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'Job', 'type': 'string', 'nullable': True, 'metadata': {}},\n",
       "     {'name': 'Dependents',\n",
       "      'type': 'integer',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'Telephone', 'type': 'string', 'nullable': True, 'metadata': {}},\n",
       "     {'name': 'ForeignWorker',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}}]},\n",
       "   'training_data_schema': {'type': 'struct',\n",
       "    'id': '1',\n",
       "    'fields': [{'name': 'CheckingStatus',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'LoanDuration',\n",
       "      'type': 'integer',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'CreditHistory',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'LoanPurpose',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'LoanAmount',\n",
       "      'type': 'integer',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'ExistingSavings',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'EmploymentDuration',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'InstallmentPercent',\n",
       "      'type': 'integer',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'Sex', 'type': 'string', 'nullable': True, 'metadata': {}},\n",
       "     {'name': 'OthersOnLoan',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'CurrentResidenceDuration',\n",
       "      'type': 'integer',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'OwnsProperty',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'Age', 'type': 'integer', 'nullable': True, 'metadata': {}},\n",
       "     {'name': 'InstallmentPlans',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'Housing', 'type': 'string', 'nullable': True, 'metadata': {}},\n",
       "     {'name': 'ExistingCreditsCount',\n",
       "      'type': 'integer',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'Job', 'type': 'string', 'nullable': True, 'metadata': {}},\n",
       "     {'name': 'Dependents',\n",
       "      'type': 'integer',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'Telephone', 'type': 'string', 'nullable': True, 'metadata': {}},\n",
       "     {'name': 'ForeignWorker',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {}},\n",
       "     {'name': 'Risk',\n",
       "      'type': 'string',\n",
       "      'nullable': True,\n",
       "      'metadata': {'modeling_role': 'target'}}]}}}}"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pre_prod_asset_details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "id": "67443902-29ae-4f39-a0af-369e8c2b7ab1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "61003e50-20d6-4ce8-a379-013a03c44229\n"
     ]
    }
   ],
   "source": [
    "pre_prod_subscription = wos_client.subscriptions.add(\n",
    "        data_mart_id=data_mart_id,\n",
    "        service_provider_id=service_provider_id,\n",
    "        asset=pre_prod_asset,\n",
    "        deployment=pre_prod_asset_deployment,\n",
    "        asset_properties=pre_prod_asset_properties_request).result\n",
    "pre_prod_subscription_id = pre_prod_subscription.metadata.id\n",
    "print(pre_prod_subscription_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "id": "92e408de-7e75-4df5-9ea9-20c43253bfc5"
   },
   "outputs": [],
   "source": [
    "# Create challenger subscription\n",
    "challenger_deployment = wml_client.deployments.get_details(deployment_uid=challenger_deployment_uid)\n",
    "challenger_asset_details=wos_client.service_providers.get_deployment_asset(data_mart_id=data_mart_id,service_provider_id=service_provider_id,deployment_id=challenger_deployment_uid,deployment_space_id=space_id)\n",
    "challenger_asset = Asset(\n",
    "    asset_id=challenger_asset_details[\"entity\"][\"asset\"][\"asset_id\"],\n",
    "    name=challenger_asset_details[\"entity\"][\"asset\"][\"name\"],\n",
    "    url=challenger_asset_details[\"entity\"][\"asset\"][\"url\"],\n",
    "    asset_type=AssetTypes.MODEL,\n",
    "    input_data_type=InputDataType.STRUCTURED,\n",
    "    problem_type=ProblemType.BINARY_CLASSIFICATION,\n",
    "    model_type='scikit-learn_1.1'\n",
    ")\n",
    "challenger_asset_deployment = AssetDeploymentRequest(\n",
    "    deployment_id=challenger_deployment_uid,\n",
    "    name=PRE_PROD_CHALLENGER_DEPLOYMENT_NAME,\n",
    "    deployment_type=DeploymentTypes.ONLINE,\n",
    "    url=challenger_deployment[\"entity\"][\"status\"][\"online_url\"][\"url\"]\n",
    ")\n",
    "\n",
    "challenger_asset_properties_request = AssetPropertiesRequest(\n",
    "    label_column=\"Risk\",\n",
    "    probability_fields=[\"probability\"],\n",
    "    prediction_field=\"prediction\",\n",
    "    feature_fields=[\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"],\n",
    "    categorical_fields=[\"CheckingStatus\",\"CreditHistory\",\"LoanPurpose\",\"ExistingSavings\",\"EmploymentDuration\",\"Sex\",\"OthersOnLoan\",\"OwnsProperty\",\"InstallmentPlans\",\"Housing\",\"Job\",\"Telephone\",\"ForeignWorker\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "id": "951027ef14c94d659d39fdc571d3ddbc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c0174a6c-d8e6-458b-8549-cd03d7df486e\n"
     ]
    }
   ],
   "source": [
    "challenger_subscription = wos_client.subscriptions.add(\n",
    "        data_mart_id=data_mart_id,\n",
    "        service_provider_id=service_provider_id,\n",
    "        asset=challenger_asset,\n",
    "        deployment=challenger_asset_deployment,\n",
    "        asset_properties=challenger_asset_properties_request).result\n",
    "challenger_subscription_id = challenger_subscription.metadata.id\n",
    "print(challenger_subscription_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "id": "dbd05839-fac8-4f66-8800-d36ec5d31ac7"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Subscriptions</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>asset_id</th><th style='border: 1px solid #dddddd'>asset_type</th><th style='border: 1px solid #dddddd'>asset_name</th><th style='border: 1px solid #dddddd'>data_mart_id</th><th style='border: 1px solid #dddddd'>deployment_id</th><th style='border: 1px solid #dddddd'>deployment_name</th><th style='border: 1px solid #dddddd'>service_provider_id</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>67e11064-81b2-4ea8-addf-b39c218299b4</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>German Credit Risk Model - Challenger</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>978e26e1-509c-4e03-b20a-baacfbbe222b</td><td style='border: 1px solid #dddddd'>German Credit Risk Model - Challenger</td><td style='border: 1px solid #dddddd'>81655d73-6ea3-44ae-9876-86df1fa6390d</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-08-07 06:07:38.473000+00:00</td><td style='border: 1px solid #dddddd'>c0174a6c-d8e6-458b-8549-cd03d7df486e</td></tr><tr><td style='border: 1px solid #dddddd'>f1b4fe35-1a75-4dca-a9b3-bc5dd7ed43db</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>German Credit Risk Model - PreProd</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>42b13344-baa1-45a0-89ca-ef662387f691</td><td style='border: 1px solid #dddddd'>German Credit Risk Model - PreProd</td><td style='border: 1px solid #dddddd'>81655d73-6ea3-44ae-9876-86df1fa6390d</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-08-07 06:07:24.153000+00:00</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>a4a1a71d-2a44-43f8-b92a-ba3c80dc36fd</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Credit Risk python Fn Model</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>90b19695-de03-41b9-b820-2d3e11c2f662</td><td style='border: 1px solid #dddddd'>dep_Credit Risk python Fn Model</td><td style='border: 1px solid #dddddd'>e413a76d-802c-44f3-8c55-300c5aa2659b</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-08-06 13:32:53.985000+00:00</td><td style='border: 1px solid #dddddd'>77c99979-8408-40b7-ab3a-e3c0a8a80aab</td></tr><tr><td style='border: 1px solid #dddddd'>eed8b225-19e5-4460-9b6f-7271dc1e3ff2</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Adult Census Income Classifier Model</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>038b8c68-6615-4c0d-8d46-15783d6ab9c8</td><td style='border: 1px solid #dddddd'>Adult Census Income Classifier Deployment</td><td style='border: 1px solid #dddddd'>c3601bc6-22c9-45a1-b016-c29cf0bf64b4</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-08-06 13:26:09.549000+00:00</td><td style='border: 1px solid #dddddd'>c274e76c-6bb9-43ce-ba64-127a3db95ede</td></tr><tr><td style='border: 1px solid #dddddd'>438ca544-9bd1-48c2-8e8d-3de4ef4ca79b</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>WML_IAE4</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>78a0af9e-1014-4fb1-b22a-5e11f4fd70e7</td><td style='border: 1px solid #dddddd'>WML_IAE4</td><td style='border: 1px solid #dddddd'>d90c6bf2-49c6-4179-9876-8b85b0247d95</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-02 07:03:31.504000+00:00</td><td style='border: 1px solid #dddddd'>e34b9b87-b6e1-4c53-b92e-cb80dea042be</td></tr><tr><td style='border: 1px solid #dddddd'>592b902d-3dc9-4e56-8bcb-86cbf1a6d8a9</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>gcr - P2 XGB Classifier - Model</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>2b976af0-e4ab-4859-af7d-2f2287d864ad</td><td style='border: 1px solid #dddddd'>gcr model</td><td style='border: 1px solid #dddddd'>4d2f2fb2-6b64-4d58-8f13-257166e468e9</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-17 07:11:44.727000+00:00</td><td style='border: 1px solid #dddddd'>e2df4ec7-6c75-416f-a444-8d21389f7513</td></tr><tr><td style='border: 1px solid #dddddd'>e3ac9fc3-bccf-4a4e-b37b-490bfb93dd81</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>GCR AutoAI - P2 XGB Classifier - Model</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>6399e6e8-df5a-4370-9af4-34b2f2e76bc6</td><td style='border: 1px solid #dddddd'>GCR Auto AI</td><td style='border: 1px solid #dddddd'>a7ca157a-de07-457a-8c4c-b1a2e998699c</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-03 10:37:20.487000+00:00</td><td style='border: 1px solid #dddddd'>ce36911c-75f6-4f99-b4d2-b71e5d55a802</td></tr><tr><td style='border: 1px solid #dddddd'>592b902d-3dc9-4e56-8bcb-86cbf1a6d8a9</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>gcr - P2 XGB Classifier - Model</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>2b976af0-e4ab-4859-af7d-2f2287d864ad</td><td style='border: 1px solid #dddddd'>gcr model</td><td style='border: 1px solid #dddddd'>4d2f2fb2-6b64-4d58-8f13-257166e468e9</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-02 15:33:17.454000+00:00</td><td style='border: 1px solid #dddddd'>e96278a6-7190-48f3-b8bd-945fa48cfe50</td></tr><tr><td style='border: 1px solid #dddddd'>327d8aea-ecfc-4990-9bb9-601a1695094d</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>GCR AutoAI - P2 XGB Classifier - Model</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>755c3e75-24b5-4839-8a8f-3f85c07a40c9</td><td style='border: 1px solid #dddddd'>GCR demo</td><td style='border: 1px solid #dddddd'>a7ca157a-de07-457a-8c4c-b1a2e998699c</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-02 12:03:14.224000+00:00</td><td style='border: 1px solid #dddddd'>a0f86241-8bfc-4322-895a-2597512e1653</td></tr><tr><td style='border: 1px solid #dddddd'>b21904ef-7478-4dae-b93f-c120e95c9200</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>My SDK Batch Subscription-db2</td><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>a10a121b-2394-4fe7-9a2d-f0520abd212c</td><td style='border: 1px solid #dddddd'>My SDK Batch Subscription-db2</td><td style='border: 1px solid #dddddd'>644befcd-6d36-4f4d-a30a-cd51a28b63fe</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-02 09:04:38.883000+00:00</td><td style='border: 1px solid #dddddd'>70c9c394-2530-4c0f-b0b2-6b81e44612d0</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: First 10 records were displayed.\n"
     ]
    }
   ],
   "source": [
    "wos_client.subscriptions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "id": "b123565b-aa22-4e18-b601-25817ebff9f0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'61003e50-20d6-4ce8-a379-013a03c44229'"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pre_prod_subscription_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "id": "d51b3056-6e8e-4914-a98f-958a53c00210"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'c0174a6c-d8e6-458b-8549-cd03d7df486e'"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "challenger_subscription_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5d16d0e9-9d15-47d3-9001-e7e25f8d8420"
   },
   "source": [
    "## Patch the training data reference in both the preprod & challenger subscription\n",
    "\n",
    "Please load the the training data from https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/german_credit_data_biased_training.csv to a table in the DB2 that comes along with CPD. Refer that training data schema and table in the below cell as the training data reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "id": "c5dd05f6-5b6a-424c-8596-5f0d1fca403c"
   },
   "outputs": [],
   "source": [
    "headers = {}\n",
    "headers[\"Content-Type\"] = \"application/json\"\n",
    "headers[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n",
    "\n",
    "training_data_reference = {\n",
    "  \"connection\": {\n",
    "    \"connection_string\": \"jdbc:db2://dashdb-txn-sbox-yp-xxxx-xx.xxxx.dal.bluemix.net:50000/BLUDB\",\n",
    "    \"database_name\": \"BLUDB\",\n",
    "    \"hostname\": \"dashdb-txn-sbox-yp-xxxx-xx.xxxx.dal.bluemix.net\",\n",
    "    \"password\": \"pxvb@xxxxxxx\",\n",
    "    \"username\": \"tkqxxxxx\"\n",
    "  },\n",
    "  \"location\": {\n",
    "    \"schema_name\": \"TKQXXXXXX\",\n",
    "    \"table_name\": \"CREDIT_RISK_TRAIN_DATA\"\n",
    "  },\n",
    "  \"name\": \"German credit risk training data\",\n",
    "  \"type\": \"db2\"\n",
    "}\n",
    "\n",
    "payload = [\n",
    " {\n",
    "   \"op\": \"replace\",\n",
    "   \"path\": \"/asset_properties/training_data_reference\",\n",
    "   \"value\": training_data_reference\n",
    " }\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "id": "5f8350cd073a41078c1d49c0afcf06df"
   },
   "outputs": [],
   "source": [
    "response = wos_client.subscriptions.update(pre_prod_subscription_id, payload)\n",
    "response.result.to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "id": "a90a8863-d279-48af-8a91-272c6c8ea102"
   },
   "outputs": [],
   "source": [
    "response = wos_client.subscriptions.update(challenger_subscription_id, payload)\n",
    "response.result.to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "75289b15-faeb-4dfe-a420-1bd3e7fb60e1"
   },
   "source": [
    "### Score the model so we can configure monitors\n",
    "Now that the WML service has been bound and the subscription has been created, we need to send a request to the model before we configure OpenScale. This allows OpenScale to create a payload log in the datamart with the correct schema, so it can capture data coming into and out of the model. First, the code gets the model deployment's endpoint URL, and then sends a few records for predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "id": "41491ac0-c360-42ec-9f55-4cd0c5e32407"
   },
   "outputs": [],
   "source": [
    "fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\n",
    "values = [\n",
    "  [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n",
    "  [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n",
    "]\n",
    "\n",
    "payload_scoring = {\"fields\": fields,\"values\": values}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "id": "eacd013d-8e52-430e-9c9f-3fb42a4add4d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Single record Scoring result: \n",
      " fields: ['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker', 'CheckingStatus_IX', 'CreditHistory_IX', 'EmploymentDuration_IX', 'ExistingSavings_IX', 'ForeignWorker_IX', 'Housing_IX', 'InstallmentPlans_IX', 'Job_IX', 'LoanPurpose_IX', 'OthersOnLoan_IX', 'OwnsProperty_IX', 'Sex_IX', 'Telephone_IX', 'features', 'rawPrediction', 'probability', 'prediction', 'predictedLabel'] \n",
      " values:  ['no_checking', 13, 'credits_paid_to_date', 'car_new', 1343, '100_to_500', '1_to_4', 2, 'female', 'none', 3, 'savings_insurance', 46, 'none', 'own', 2, 'skilled', 1, 'none', 'yes', 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, [21, [1, 3, 5, 13, 14, 15, 16, 17, 18, 19, 20], [1.0, 1.0, 1.0, 13.0, 1343.0, 2.0, 3.0, 13.0, 46.0, 2.0, 1.0]], [0.37923501094948475, -0.37923501094948475], [0.6810214668555962, 0.31897853314440383], 0.0, 'No Risk']\n"
     ]
    }
   ],
   "source": [
    "payload = {\n",
    "    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n",
    "}\n",
    "scoring_response = wml_client.deployments.score(pre_prod_deployment_uid, payload)\n",
    "\n",
    "print('Single record Scoring result:', '\\n fields:', scoring_response['predictions'][0]['fields'], '\\n values: ', scoring_response['predictions'][0]['values'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "id": "1e48deef-7631-4ee1-8ec4-d15f10313c59"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Payload data set id:  0da980a0-4be1-4bd8-85b7-8cf5b694d5a1\n",
      "Number of records in the payload logging table: 8\n"
     ]
    }
   ],
   "source": [
    "time.sleep(10)\n",
    "pre_prod_pl_data_set_id = None\n",
    "pre_prod_pl_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, \n",
    "                                                target_target_id=pre_prod_subscription_id, \n",
    "                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id\n",
    "if pre_prod_pl_data_set_id is None:\n",
    "    print(\"Payload data set not found. Please check subscription status.\")\n",
    "else:\n",
    "    print(\"Payload data set id: \", pre_prod_pl_data_set_id)\n",
    "    pl_records_count = wos_client.data_sets.get_records_count(pre_prod_pl_data_set_id)\n",
    "    print(\"Number of records in the payload logging table: {}\".format(pl_records_count))\n",
    "    if pl_records_count == 0:\n",
    "        print(\"Automatic payload logging did not happen, performing explicit payload logging.\")\n",
    "        wos_client.data_sets.store_records(data_set_id=pre_prod_pl_data_set_id, request_body=[PayloadRecord(\n",
    "                       scoring_id=\"mrm_\"+str(uuid.uuid4()),\n",
    "                       request=payload_scoring,\n",
    "                       response=scoring_response['predictions'][0]\n",
    "                   )])\n",
    "        time.sleep(5)\n",
    "        pl_records_count = wos_client.data_sets.get_records_count(pre_prod_pl_data_set_id)\n",
    "        print(\"Number of records in the payload logging table: {}\".format(pl_records_count))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "id": "b3360e3b-a588-4b14-9dff-ab766209c501"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Single record Scoring result: \n",
      " fields: ['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker', 'CheckingStatus_IX', 'CreditHistory_IX', 'EmploymentDuration_IX', 'ExistingSavings_IX', 'ForeignWorker_IX', 'Housing_IX', 'InstallmentPlans_IX', 'Job_IX', 'LoanPurpose_IX', 'OthersOnLoan_IX', 'OwnsProperty_IX', 'Sex_IX', 'Telephone_IX', 'features', 'rawPrediction', 'probability', 'prediction', 'predictedLabel'] \n",
      " values:  ['no_checking', 13, 'credits_paid_to_date', 'car_new', 1343, '100_to_500', '1_to_4', 2, 'female', 'none', 3, 'savings_insurance', 46, 'none', 'own', 2, 'skilled', 1, 'none', 'yes', 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, [21, [1, 3, 5, 13, 14, 15, 16, 17, 18, 19, 20], [1.0, 1.0, 1.0, 13.0, 1343.0, 2.0, 3.0, 13.0, 46.0, 2.0, 1.0]], [0.37923501094948475, -0.37923501094948475], [0.6810214668555962, 0.31897853314440383], 0.0, 'No Risk']\n"
     ]
    }
   ],
   "source": [
    "payload = {\n",
    "    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n",
    "}\n",
    "scoring_response = wml_client.deployments.score(challenger_deployment_uid, payload)\n",
    "\n",
    "print('Single record Scoring result:', '\\n fields:', scoring_response['predictions'][0]['fields'], '\\n values: ', scoring_response['predictions'][0]['values'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "id": "1f7b180e-46e0-4e14-90de-0b4f9a701a02"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Payload data set id:  5a5e5126-d80b-4538-b7aa-c09545855da8\n",
      "Number of records in the payload logging table: 8\n"
     ]
    }
   ],
   "source": [
    "from ibm_watson_openscale.supporting_classes import *\n",
    "time.sleep(10)\n",
    "challenger_pl_data_set_id = None\n",
    "challenger_pl_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, \n",
    "                                                target_target_id=challenger_subscription_id, \n",
    "                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id\n",
    "if challenger_pl_data_set_id is None:\n",
    "    print(\"Payload data set not found. Please check subscription status.\")\n",
    "else:\n",
    "    print(\"Payload data set id: \", challenger_pl_data_set_id)\n",
    "    pl_records_count = wos_client.data_sets.get_records_count(challenger_pl_data_set_id)\n",
    "    print(\"Number of records in the payload logging table: {}\".format(pl_records_count))\n",
    "    if pl_records_count == 0:\n",
    "        print(\"Automatic payload logging did not happen, performing explicit payload logging.\")\n",
    "        wos_client.data_sets.store_records(data_set_id=challenger_pl_data_set_id, request_body=[PayloadRecord(\n",
    "                       scoring_id=\"mrm_\"+str(uuid.uuid4()),\n",
    "                       request=payload_scoring,\n",
    "                       response=scoring_response['predictions'][0]\n",
    "                   )])\n",
    "        time.sleep(5)\n",
    "        pl_records_count = wos_client.data_sets.get_records_count(challenger_pl_data_set_id)\n",
    "        print(\"Number of records in the payload logging table: {}\".format(pl_records_count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9f743917-8bd6-4b0b-a7a4-22e323f9ca0f"
   },
   "source": [
    "# Quality monitoring\n",
    "\n",
    "## Enable quality monitoring\n",
    "The code below waits ten seconds to allow the payload logging table to be set up before it begins enabling monitors. First, it turns on the quality (accuracy) monitor and sets an alert threshold of 80%. OpenScale will show an alert on the dashboard if the model accuracy measurement (area under the curve, in the case of a binary classifier) falls below this threshold.\n",
    "\n",
    "The second paramater supplied, min_records, specifies the minimum number of feedback records OpenScale needs before it calculates a new measurement. The quality monitor runs hourly, but the accuracy reading in the dashboard will not change until an additional 50 feedback records have been added, via the user interface, the Python client, or the supplied feedback endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "id": "951aeb7a-ea55-4391-9406-a3008b6078f5",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation 85319c11-608f-4625-b38a-1b51c661c4fc \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "preparing\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "time.sleep(10)\n",
    "target = Target(\n",
    "        target_type=TargetTypes.SUBSCRIPTION,\n",
    "        target_id=pre_prod_subscription_id\n",
    ")\n",
    "parameters = {\n",
    "    \"min_feedback_data_size\": 100,\n",
    "    \"threshold\": 0.8\n",
    "}\n",
    "pre_prod_quality_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.QUALITY.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "id": "d285aae9-f0c4-4de0-abad-beebe4472f56"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation 7f8c5ebc-4d43-41f9-b7f9-5aed8c6da04d \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "preparing\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "time.sleep(10)\n",
    "target = Target(\n",
    "        target_type=TargetTypes.SUBSCRIPTION,\n",
    "        target_id=challenger_subscription_id\n",
    ")\n",
    "parameters = {\n",
    "    \"min_feedback_data_size\": 100,\n",
    "    \"threshold\": 0.8\n",
    "}\n",
    "challenger_quality_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.QUALITY.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "13e15202-a809-4dea-81ef-29307bdac80e"
   },
   "source": [
    "# Fairness, drift monitoring and explanations \n",
    "\n",
    "## Fairness configuration\n",
    "The code below configures fairness monitoring for our model. It turns on monitoring for two features, Sex and Age. In each case, we must specify:\n",
    "\n",
    "Which model feature to monitor\n",
    "One or more majority groups, which are values of that feature that we expect to receive a higher percentage of favorable outcomes\n",
    "One or more minority groups, which are values of that feature that we expect to receive a higher percentage of unfavorable outcomes\n",
    "The threshold at which we would like OpenScale to display an alert if the fairness measurement falls below (in this case, 80%)\n",
    "Additionally, we must specify which outcomes from the model are favourable outcomes, and which are unfavourable. We must also provide the number of records OpenScale will use to calculate the fairness score. In this case, OpenScale's fairness monitor will run hourly, but will not calculate a new fairness rating until at least 100 records have been added. Finally, to calculate fairness, OpenScale must perform some calculations on the training data, so we provide the dataframe containing the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "id": "dc3be33f-48e0-4be9-b52b-660f2a0438ca"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation 5bd1947b-f9df-42ba-82d2-ebafcbe5eb42 \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'5bd1947b-f9df-42ba-82d2-ebafcbe5eb42'"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=pre_prod_subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "    \"features\": [\n",
    "        {\"feature\": \"Sex\",\n",
    "         \"majority\": ['male'],\n",
    "         \"minority\": ['female']\n",
    "         },\n",
    "        {\"feature\": \"Age\",\n",
    "         \"majority\": [[26, 75]],\n",
    "         \"minority\": [[18, 25]]\n",
    "         }\n",
    "    ],\n",
    "    \"favourable_class\": [\"No Risk\"],\n",
    "    \"unfavourable_class\": [\"Risk\"],\n",
    "    \"min_records\": 100\n",
    "}\n",
    "thresholds = [{\n",
    "    \"metric_id\": \"fairness_value\",\n",
    "    \"specific_values\": [{\n",
    "            \"applies_to\": [{\n",
    "                \"key\": \"feature\",\n",
    "                \"type\": \"tag\",\n",
    "                \"value\": \"Age\"\n",
    "            }],\n",
    "            \"value\": 95\n",
    "        },\n",
    "        {\n",
    "            \"applies_to\": [{\n",
    "                \"key\": \"feature\",\n",
    "                \"type\": \"tag\",\n",
    "                \"value\": \"Sex\"\n",
    "            }],\n",
    "            \"value\": 95\n",
    "        }\n",
    "    ],\n",
    "    \"type\": \"lower_limit\",\n",
    "    \"value\": 80.0\n",
    "}]\n",
    "\n",
    "pre_prod_fairness_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.FAIRNESS.ID,\n",
    "    target=target,\n",
    "    parameters=parameters,\n",
    "    thresholds=thresholds).result\n",
    "pre_prod_fairness_monitor_details.metadata.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "id": "0f7fef90-5b12-42dd-971f-602b1808054d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation 0f7fef90-5b12-42dd-971f-602b1808054d \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "preparing\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=challenger_subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "}\n",
    "thresholds = []\n",
    "\n",
    "challenger_fairness_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.FAIRNESS.ID,\n",
    "    target=target,\n",
    "    parameters=parameters,\n",
    "    thresholds=thresholds).result\n",
    "challenger_fairness_monitor_details.metadata.id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6626044b-78a6-4788-beb3-93b83c1adb9b"
   },
   "source": [
    "## Drift configuration\n",
    "\n",
    "Enable the drift configuration for both the subscription created with a threshold of 10% and minimal sample as 100 records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "id": "2d39d4ac-31eb-4855-88bb-afb23c7f4c39"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleted existing drift monitor instance with id:  2a5eaaf0-4681-4d3d-81a2-4f243f66932b\n",
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation b43d2e79-e244-44c2-9dc7-c5a7bae56a2e \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "preparing\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'b43d2e79-e244-44c2-9dc7-c5a7bae56a2e'"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "monitor_instances = wos_client.monitor_instances.list().result.monitor_instances\n",
    "for monitor_instance in monitor_instances:\n",
    "    monitor_def_id=monitor_instance.entity.monitor_definition_id\n",
    "    if monitor_def_id == wos_client.monitor_definitions.MONITORS.DRIFT.ID and monitor_instance.entity.target.target_id == pre_prod_subscription_id:\n",
    "        wos_client.monitor_instances.delete(monitor_instance.metadata.id)\n",
    "        print('Deleted existing drift monitor instance with id: ', monitor_instance.metadata.id)\n",
    "\n",
    "\n",
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=pre_prod_subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "    \"min_samples\": 100,\n",
    "    \"train_drift_model\": True,\n",
    "    \"enable_model_drift\": False,\n",
    "    \"enable_data_drift\": True\n",
    "}\n",
    "\n",
    "pre_prod_drift_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "pre_prod_drift_monitor_details.metadata.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "id": "3351a278-ea8e-4767-b266-d69aca03028b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation 3d6fba63-767a-4813-b34d-964c37927ba6 \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "preparing\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'3d6fba63-767a-4813-b34d-964c37927ba6'"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "monitor_instances = wos_client.monitor_instances.list().result.monitor_instances\n",
    "for monitor_instance in monitor_instances:\n",
    "    monitor_def_id=monitor_instance.entity.monitor_definition_id\n",
    "    if monitor_def_id == wos_client.monitor_definitions.MONITORS.DRIFT.ID and monitor_instance.entity.target.target_id == challenger_subscription_id:\n",
    "        wos_client.monitor_instances.delete(monitor_instance.metadata.id)\n",
    "        print('Deleted existing drift monitor instance with id: ', monitor_instance.metadata.id)\n",
    "\n",
    "\n",
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=challenger_subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "    \"min_samples\": 100,\n",
    "    \"train_drift_model\": True,\n",
    "    \"enable_model_drift\": False,\n",
    "    \"enable_data_drift\": True\n",
    "}\n",
    "\n",
    "challenger_drift_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "challenger_drift_monitor_details.metadata.id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Drift V2 Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation d6b3f5bc-82f5-4042-b6b2-3ac1aab9eb07 \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "preparing........\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'d6b3f5bc-82f5-4042-b6b2-3ac1aab9eb07'"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from random import choices\n",
    "\n",
    "monitor_instances = wos_client.monitor_instances.list().result.monitor_instances\n",
    "for monitor_instance in monitor_instances:\n",
    "    monitor_def_id=monitor_instance.entity.monitor_definition_id\n",
    "    if monitor_def_id == wos_client.monitor_definitions.MONITORS.DRIFT_v2.ID and monitor_instance.entity.target.target_id == pre_prod_subscription_id:\n",
    "        wos_client.monitor_instances.delete(monitor_instance.metadata.id)\n",
    "        print('Deleted existing drift monitor instance with id: ', monitor_instance.metadata.id)\n",
    "\n",
    "\n",
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=pre_prod_subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "        \"min_samples\": 10,\n",
    "        \"max_samples\": 1000,\n",
    "        \"train_archive\": True,\n",
    "        \"features\": {\n",
    "            \"fields\": fields\n",
    "            },\n",
    "        \"most_important_features\": { \n",
    "            \"fields\": choices(fields, k=5)\n",
    "        },\n",
    "    }\n",
    "\n",
    "pre_prod_drift_v2_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT_V2.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "pre_prod_drift_v2_monitor_details.metadata.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation 32a654fc-7488-47a2-be43-bbf325e5cfb7 \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "preparing\n",
      "error\n",
      "\n",
      "-----------------------------------------------------\n",
      " Monitor instance creation failed with status: error \n",
      "-----------------------------------------------------\n",
      "\n",
      "\n",
      "Reason: ['code: AIQDD1035E, message: Drift configuration failed. a must be greater than 0 unless no samples are taken']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'32a654fc-7488-47a2-be43-bbf325e5cfb7'"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "monitor_instances = wos_client.monitor_instances.list().result.monitor_instances\n",
    "for monitor_instance in monitor_instances:\n",
    "    monitor_def_id=monitor_instance.entity.monitor_definition_id\n",
    "    if monitor_def_id == wos_client.monitor_definitions.MONITORS.DRIFT_V2.ID and monitor_instance.entity.target.target_id == challenger_subscription_id:\n",
    "        wos_client.monitor_instances.delete(monitor_instance.metadata.id)\n",
    "        print('Deleted existing drift monitor instance with id: ', monitor_instance.metadata.id)\n",
    "\n",
    "\n",
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=challenger_subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "        \"min_samples\": 10,\n",
    "        \"max_samples\": 1000,\n",
    "        \"train_archive\": True,\n",
    "        \"features\": {\n",
    "            \"fields\": fields,\n",
    "        },\n",
    "        \"most_important_features\": { \n",
    "            \"fields\": choices(fields, k=5)\n",
    "        },\n",
    "    }\n",
    "\n",
    "challenger_drift_v2_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT_V2.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "challenger_drift_v2_monitor_details.metadata.id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cd722c63-9ac7-47f6-8929-a614056c997f"
   },
   "source": [
    "## Configure Explainability\n",
    "Finally, we provide OpenScale with the training data to enable and configure the explainability features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "id": "2c008c5b-70ff-4dbe-92b1-06e4781e5df1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation e26388e7-cf71-4564-8449-5224fa6f911f \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "preparing.\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'e26388e7-cf71-4564-8449-5224fa6f911f'"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=pre_prod_subscription_id\n",
    ")\n",
    "parameters = {\n",
    "    \"enabled\": True\n",
    "}\n",
    "pre_prod_explain_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.EXPLAINABILITY.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "pre_prod_explain_monitor_details.metadata.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "id": "58f069b8adbd4632b03f6718715c0cde"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation e26388e7-cf71-4564-8449-5224fa6f911f \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "preparing.\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'fd101aee-d9db-4b0d-b74e-eb69c64ff5f5'"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=challenger_subscription_id\n",
    ")\n",
    "parameters = {\n",
    "    \"enabled\": True\n",
    "}\n",
    "challenger_explain_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.EXPLAINABILITY.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "challenger_explain_monitor_details.metadata.id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e13c1f8b-ba92-4775-9acb-781f8796436b"
   },
   "source": [
    "## Enable model risk management (MRM) \n",
    "\n",
    "We enable the MRM configuration for both the subscriptions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "id": "c43ca07fbc8a445d86da2d96b3ec4ca8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation 23b1f186-f599-45fb-830c-2810c18bad42 \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=pre_prod_subscription_id\n",
    ")\n",
    "parameters = {\n",
    "}\n",
    "pre_prod_mrm_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id='mrm',\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "pre_prod_mrm_instance_id = pre_prod_mrm_monitor_details.metadata.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "id": "fa0b8ff607804d24bd21a27b7a7e3e3b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation d689a828-6fa7-4adb-b6e6-4ab464e866b9 \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=challenger_subscription_id\n",
    ")\n",
    "parameters = {\n",
    "}\n",
    "challenger_mrm_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id='mrm',\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "challenger_mrm_instance_id = challenger_mrm_monitor_details.metadata.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "id": "e98894c5dda44926b5c6aff1646b1005"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Monitor instances</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>data_mart_id</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>target_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>c0174a6c-d8e6-458b-8549-cd03d7df486e</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>2024-08-07 06:21:01.890000+00:00</td><td style='border: 1px solid #dddddd'>d689a828-6fa7-4adb-b6e6-4ab464e866b9</td></tr><tr><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>2024-08-07 06:20:53.482000+00:00</td><td style='border: 1px solid #dddddd'>23b1f186-f599-45fb-830c-2810c18bad42</td></tr><tr><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>1b8ebb3c-e69a-4cc1-9a0a-57424546cab3</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>model_health</td><td style='border: 1px solid #dddddd'>2024-07-01 17:17:45.382000+00:00</td><td style='border: 1px solid #dddddd'>1d0924c1-7f1a-419c-b2bc-bff1f9a5d5d0</td></tr><tr><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>error</td><td style='border: 1px solid #dddddd'>c0174a6c-d8e6-458b-8549-cd03d7df486e</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>explainability</td><td style='border: 1px solid #dddddd'>2024-08-07 06:19:13.573000+00:00</td><td style='border: 1px solid #dddddd'>fd101aee-d9db-4b0d-b74e-eb69c64ff5f5</td></tr><tr><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>explainability</td><td style='border: 1px solid #dddddd'>2024-08-07 06:18:55.727000+00:00</td><td style='border: 1px solid #dddddd'>e26388e7-cf71-4564-8449-5224fa6f911f</td></tr><tr><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>error</td><td style='border: 1px solid #dddddd'>c0174a6c-d8e6-458b-8549-cd03d7df486e</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>2024-08-07 06:18:43.293000+00:00</td><td style='border: 1px solid #dddddd'>32a654fc-7488-47a2-be43-bbf325e5cfb7</td></tr><tr><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>7c0c6db0-0c7f-415b-85f1-cae28daded5b</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>2024-06-14 03:16:37.493000+00:00</td><td style='border: 1px solid #dddddd'>87fce793-9360-4782-8b02-1f83bd4b8b2f</td></tr><tr><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>7c0c6db0-0c7f-415b-85f1-cae28daded5b</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>model_health</td><td style='border: 1px solid #dddddd'>2024-06-14 03:16:37.532000+00:00</td><td style='border: 1px solid #dddddd'>bf46efe7-3db2-4627-92db-e4119cb9bbe0</td></tr><tr><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>2024-08-07 06:17:45.589000+00:00</td><td style='border: 1px solid #dddddd'>d6b3f5bc-82f5-4042-b6b2-3ac1aab9eb07</td></tr><tr><td style='border: 1px solid #dddddd'>00000000-0000-0000-0000-000000000000</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>c0174a6c-d8e6-458b-8549-cd03d7df486e</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>drift</td><td style='border: 1px solid #dddddd'>2024-08-07 06:17:22.268000+00:00</td><td style='border: 1px solid #dddddd'>3d6fba63-767a-4813-b34d-964c37927ba6</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: First 10 records were displayed.\n"
     ]
    }
   ],
   "source": [
    "wos_client.monitor_instances.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6a050eb6-b070-4789-af86-0602cb1af95a"
   },
   "source": [
    "## Create test data sets from the training data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "id": "76a4364f-caf8-46fc-8b21-b5c8d99f71d4"
   },
   "outputs": [],
   "source": [
    "test_data_1 = pd_data[1:201]\n",
    "test_data_1.to_csv(\"german_credit_risk_test_data_1.csv\", encoding=\"utf-8\", index=False)\n",
    "test_data_2 = pd_data[201:401]\n",
    "test_data_2.to_csv(\"german_credit_risk_test_data_2.csv\", encoding=\"utf-8\", index=False)\n",
    "test_data_3 = pd_data[401:601]\n",
    "test_data_3.to_csv(\"german_credit_risk_test_data_3.csv\", encoding=\"utf-8\", index=False)\n",
    "test_data_4 = pd_data[601:801]\n",
    "test_data_4.to_csv(\"german_credit_risk_test_data_4.csv\", encoding=\"utf-8\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9a859d02-9000-4640-875e-75e21f631919"
   },
   "source": [
    "## Function to upload, evaluate and check the status of the evaluation\n",
    "\n",
    "This function will upload the test data CSV and trigger the risk evaluation. It will iterate and check the status of the evaluation until its finished with a finite wait duration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "id": "bfccd770-08e0-492e-87c2-b3b0cc73ad58"
   },
   "outputs": [],
   "source": [
    "def upload_and_evaluate(file_name, mrm_instance_id):\n",
    "    \n",
    "    print(\"Running upload and evaluate for {}\".format(file_name))\n",
    "    import json\n",
    "    import time\n",
    "    from datetime import datetime\n",
    "\n",
    "    status = None\n",
    "    monitoring_run_id = None\n",
    "    GET_UPLOAD_AND_EVALUATION_STATUS_RETRIES = 32\n",
    "    GET_UPLOAD_AND_EVALUATION_STATUS_INTERVAL = 10\n",
    "    \n",
    "    if file_name is not None:\n",
    "      \n",
    "        response = wos_client.monitor_instances.mrm.evaluate_risk(\n",
    "                        monitor_instance_id = mrm_instance_id,\n",
    "                        test_data_set_name = file_name,\n",
    "                        feedback_data_path = file_name,\n",
    "                        test_data_path=file_name)\n",
    "    \n",
    "        if response.status_code != 202:\n",
    "            print(\"Upload and evalaute for {0} failed with error : {1}\".format(file_name, response.status_code))\n",
    "            return\n",
    "    \n",
    "        for i in range(GET_UPLOAD_AND_EVALUATION_STATUS_RETRIES):\n",
    "        \n",
    "            response = wos_client.monitor_instances.mrm.get_risk_evaluation(\n",
    "                            monitor_instance_id = mrm_instance_id)\n",
    "            \n",
    "            if response.status_code != 200:\n",
    "                print(\"Getting status of upload and evalaute for {0} failed with error : {1}\".format(file_name, response.status_code))\n",
    "                return\n",
    "            \n",
    "            response = response.result._to_dict()\n",
    "            \n",
    "            if \"metadata\" in response and \"id\" in response[\"metadata\"]:\n",
    "                monitoring_run_id = response[\"metadata\"][\"id\"]\n",
    "            if \"entity\" in response and \"status\" in response[\"entity\"]:\n",
    "                status = response[\"entity\"][\"status\"][\"state\"]\n",
    "            \n",
    "            if status is not None:\n",
    "                print(datetime.utcnow().strftime('%H:%M:%S'), status.lower())\n",
    "                if status.lower() in [\"finished\", \"completed\"]:\n",
    "                    break\n",
    "                elif \"error\" in status.lower():\n",
    "                    print(response)\n",
    "                    break\n",
    "\n",
    "            time.sleep(GET_UPLOAD_AND_EVALUATION_STATUS_INTERVAL)\n",
    "\n",
    "    return status, monitoring_run_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "85dfeacb-52c8-4380-bfe7-08d6d28eff56"
   },
   "source": [
    "## Perform Risk Evaluations\n",
    "\n",
    "We now start performing evaluations of smaller data sets against both the PreProd and Challenger subscriptions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "id": "cc2fe2df-19f4-41f0-9a80-e746e397ed8a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Running upload and evaluate for german_credit_risk_test_data_1.csv\n",
      "15:34:06 upload_in_progress\n",
      "15:34:16 upload_in_progress\n",
      "15:34:26 upload_in_progress\n",
      "15:34:36 finished\n"
     ]
    }
   ],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_1.csv\", pre_prod_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "id": "71cefcf8-2c60-45a9-87ef-d23279fc06b3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Running upload and evaluate for german_credit_risk_test_data_2.csv\n",
      "15:35:03 upload_in_progress\n",
      "15:35:13 upload_in_progress\n",
      "15:35:23 finished\n"
     ]
    }
   ],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_2.csv\", pre_prod_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "id": "5cbc1cb3-08cb-4b04-923a-3ebaec929438"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Running upload and evaluate for german_credit_risk_test_data_3.csv\n",
      "15:41:12 upload_in_progress\n",
      "15:41:22 upload_in_progress\n",
      "15:41:33 upload_in_progress\n",
      "15:41:43 upload_in_progress\n",
      "15:41:53 finished\n"
     ]
    }
   ],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_3.csv\", pre_prod_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "id": "7e4d0ee0-4452-49b8-a7f1-cce3ac41cb8f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Running upload and evaluate for german_credit_risk_test_data_4.csv\n",
      "15:42:16 upload_in_progress\n",
      "15:42:26 upload_in_progress\n",
      "15:42:36 upload_in_progress\n",
      "15:42:46 upload_in_progress\n",
      "15:42:56 finished\n"
     ]
    }
   ],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_4.csv\", pre_prod_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "id": "6c44912b-05c9-4e3b-ad3f-f56bb982f88d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Running upload and evaluate for german_credit_risk_test_data_1.csv\n",
      "15:43:01 upload_in_progress\n",
      "15:43:11 upload_in_progress\n",
      "15:43:21 upload_in_progress\n",
      "15:43:31 finished\n"
     ]
    }
   ],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_1.csv\", challenger_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "id": "217941be-48cb-4be7-8b35-3ed2f9bf6f01"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Running upload and evaluate for german_credit_risk_test_data_2.csv\n",
      "15:44:08 upload_in_progress\n",
      "15:44:18 upload_in_progress\n",
      "15:44:28 upload_in_progress\n",
      "15:44:38 upload_in_progress\n",
      "15:44:48 upload_in_progress\n",
      "15:44:58 finished\n"
     ]
    }
   ],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_2.csv\", challenger_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "id": "fa2f44a1-815a-428f-96ab-180f255eda64"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Running upload and evaluate for german_credit_risk_test_data_3.csv\n",
      "15:45:12 upload_in_progress\n",
      "15:45:22 upload_in_progress\n",
      "15:45:32 finished\n"
     ]
    }
   ],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_3.csv\", challenger_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "id": "2e03dd93-dfa2-4095-a1d8-0eb127cceeb6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Running upload and evaluate for german_credit_risk_test_data_4.csv\n",
      "15:45:41 upload_in_progress\n",
      "15:45:51 upload_in_progress\n",
      "15:46:01 upload_in_progress\n",
      "15:46:11 upload_in_progress\n"
     ]
    }
   ],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_4.csv\", challenger_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "426f03ef-de4a-4bc4-9756-744349f5b660"
   },
   "source": [
    "## Explore the Model Risk Management UI\n",
    "\n",
    "Here is a quick recap of what we have done so far.\n",
    "\n",
    "1. We've deployed two Credit Risk Model to a WML instance that is designated as Pre-Production\n",
    "2. We've created subscriptions of these two model deployments in OpenScale\n",
    "3. Configured all monitors supported by OpenScale for these subscriptions\n",
    "4. We've performed a few risk evaluations against both these susbscription with the same set of test data\n",
    "\n",
    "Now, please explore the Model Risk Management UI to visualize the results, compare the performance of models, download the evaluation report as PDF. For more information, refer to the Beta Guide section \"Work in Watson OpenScale.\"\n",
    "\n",
    "Link to OpenScale : https://CLUSTER_URL/aiopenscale/insights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "215eb279-37e0-4834-b34c-846b33c2700e"
   },
   "source": [
    "# Promote pre-production model to production \n",
    "\n",
    "After you have reviewed the evaluation results of the PreProd Vs Challenger and if you make the decision to promote the Challenger model to Production, the first thing you need to do is to deploy the model into a WML instance that is designated as Production instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "98667af1-fea5-4a7f-a547-179582280646"
   },
   "source": [
    "## Deploy model to production WML instance "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "id": "e4081ad4-010c-478e-b5b2-f762da9869c0"
   },
   "outputs": [],
   "source": [
    "PROD_MODEL_NAME=\"German Credit Risk Model - Prod\"\n",
    "PROD_DEPLOYMENT_NAME=\"German Credit Risk Model - Prod\"\n",
    "\n",
    "PROD_SPACE_NAME=\"prod\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "id": "4d07ab26-cf94-41f1-b5bb-c8272a5e6fe7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: 'limit' is not provided. Only first 50 records will be displayed if the number of records exceed 50\n",
      "------------------------------------  -----------------------------------------------------------------------  ------------------------\n",
      "ID                                    NAME                                                                     CREATED\n",
      "06873ce9-baf3-46a6-86c4-54c1e3d84429  tutorial_space                                                           2024-08-06T13:29:18.489Z\n",
      "e8b87647-a0e9-4932-920f-1e8f1d1f383d  tutorial-space                                                           2024-08-06T12:55:07.962Z\n",
      "832d6bbe-a2ea-451f-8d94-853da7c52b00  FrauDScore                                                               2024-08-05T07:48:21.490Z\n",
      "8d3a3e65-4875-4a5e-a6bd-617d2deeb9ac  Neelima                                                                  2024-08-05T05:42:10.490Z\n",
      "3a051b57-d5f6-4290-a2ac-5f66a6ed6d65  openscale-express-path-preprod-00000000-0000-0000-0000-1722503677398532  2024-08-01T09:56:04.456Z\n",
      "3d10da0f-497d-43d6-a8cd-71d15b833511  openscale-express-path-00000000-0000-0000-0000-1722503677398532          2024-08-01T09:55:43.160Z\n",
      "45683c18-10ea-4b25-8888-cf3c12646b8d  hk_pre_prod                                                              2024-07-27T12:34:07.223Z\n",
      "7f4afba9-1a6b-4f27-9a0a-507182adf269  goSales deployment space                                                 2024-07-09T14:42:26.982Z\n",
      "47dab2f7-5fe2-49bb-bf69-2cad9c7c5aa8  Notebook demo space                                                      2024-07-09T07:44:12.138Z\n",
      "a4bc50f8-8acf-47b5-b82d-103f311a5fa4  GCR Auto AI prod space                                                   2024-07-04T06:40:01.094Z\n",
      "4a0ad576-aad5-43ee-9457-11b4d2f39cf6  GCR AutoAI space Demo                                                    2024-07-03T14:26:17.440Z\n",
      "a8233d23-2f8b-40fc-b9a2-45aec8b53b83  openscale-express-path-preprod-00000000-0000-0000-0000-1719914010257861  2024-07-02T09:59:03.652Z\n",
      "3e3e2ae1-23d9-468d-abff-02c127d052ef  openscale-express-path-00000000-0000-0000-0000-1719914010257861          2024-07-02T09:58:32.157Z\n",
      "dcb2cb00-cb6f-47fe-8543-cffd5df0786a  notebooks CPD                                                            2024-07-01T17:10:55.025Z\n",
      "c3905865-4e73-43d1-a18f-24cf3a8b9163  wml_preprod                                                              2024-06-29T17:49:53.207Z\n",
      "7e5a8be6-9103-4c22-9c43-b66f3d8364de  poojitha_notebooks_space                                                 2024-06-29T13:53:46.645Z\n",
      "16ccd855-46bd-43ed-8219-5f00ac565d08  shreya-space                                                             2024-06-26T04:29:17.302Z\n",
      "bc3b9797-c509-4fb4-a424-f67b1e2ed4be  QUALITY_WMLV4_PREPROD                                                    2024-06-23T12:23:04.790Z\n",
      "e396e187-2977-47b4-ade3-1539f9f10adc  QUALITY_WMLV4_PROD                                                       2024-06-23T12:22:54.422Z\n",
      "40c4d032-0339-4da6-bfec-4bdb096c9650  shreya                                                                   2024-06-20T10:54:20.088Z\n",
      "088c142e-f35e-4e48-a30c-ad55a6edeecc  notebooks 5.0                                                            2024-06-13T04:42:07.336Z\n",
      "b9b3d3b4-6e26-4e16-807d-e8bf5e7d6984  MRM_WMLV4_PREPROD                                                        2024-06-12T15:49:26.571Z\n",
      "d22e2b6b-917c-4427-a40c-1a439352a742  MRM_WMLV4_PROD                                                           2024-06-12T15:49:16.185Z\n",
      "ce15e0f6-be30-4349-af47-35ae15983bf1  openscale-express-path-preprod-00000000-0000-0000-0000-000000000000      2024-06-04T05:18:51.988Z\n",
      "6264dc0e-087a-4dea-bcbc-6bd872b510fb  openscale-express-path-00000000-0000-0000-0000-000000000000              2024-06-04T05:18:30.811Z\n",
      "------------------------------------  -----------------------------------------------------------------------  ------------------------\n",
      "Space has been created. However some background setup activities might still be on-going. Check for 'status' field in the response. It has to show 'active' before space can be used. If it's not 'active', you can monitor the state with a call to spaces.get_details(space_id). Alternatively, use background_mode=False when calling client.spaces.store().\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'SUCCESS'"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wml_client.spaces.list()\n",
    "\n",
    "# Find and set the default space\n",
    "space_name=PROD_SPACE_NAME\n",
    "spaces = wml_client.spaces.get_details()['resources']\n",
    "space_id = None\n",
    "for space in spaces:\n",
    "    if space['entity']['name'] == space_name:\n",
    "        space_id = space[\"metadata\"][\"id\"]\n",
    "if space_id is None:\n",
    "    space_id = wml_client.spaces.store(\n",
    "        meta_props={wml_client.spaces.ConfigurationMetaNames.NAME: space_name})[\"metadata\"][\"id\"]\n",
    "wml_client.set.default_space(space_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "id": "24dc7643-4d5e-4003-ad6c-4b3ed347f156"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of records for training: 4485\n",
      "Number of records for evaluation: 515\n",
      "Accuracy = 0.726321\n",
      "--  ----  -------  ----  ----------  ----------------\n",
      "ID  NAME  CREATED  TYPE  SPEC_STATE  SPEC_REPLACEMENT\n",
      "--  ----  -------  ----  ----------  ----------------\n",
      "Note: Warnings!! :  Software specification spark-mllib_3.3 specified for the wml_model is deprecated and will be removed in the future. We recommend you use spark-mllib_3.4 instead. For details see Supported Frameworks https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=specifications-supported-deployment-frameworks\n",
      "{'entity': {'hybrid_pipeline_software_specs': [], 'label_column': 'Risk', 'pipeline': {'id': 'd1b44b2d-401a-4bd4-bde3-6db64dd0ff68'}, 'schemas': {'input': [{'fields': [{'metadata': {}, 'name': 'CheckingStatus', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'LoanDuration', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'CreditHistory', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'LoanPurpose', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'LoanAmount', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'ExistingSavings', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'EmploymentDuration', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'InstallmentPercent', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'Sex', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'OthersOnLoan', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'CurrentResidenceDuration', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'OwnsProperty', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'Age', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'InstallmentPlans', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'Housing', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'ExistingCreditsCount', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'Job', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'Dependents', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'Telephone', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'ForeignWorker', 'nullable': True, 'type': 'string'}], 'id': '1', 'type': 'struct'}], 'output': []}, 'software_spec': {'id': 'd11f2434-4fc7-58b7-8a62-755da64fdaf8', 'name': 'spark-mllib_3.3'}, 'training_data_references': [{'id': '1', 'location': {}, 'schema': {'fields': [{'metadata': {}, 'name': 'CheckingStatus', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'LoanDuration', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'CreditHistory', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'LoanPurpose', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'LoanAmount', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'ExistingSavings', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'EmploymentDuration', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'InstallmentPercent', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'Sex', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'OthersOnLoan', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'CurrentResidenceDuration', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'OwnsProperty', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'Age', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'InstallmentPlans', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'Housing', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'ExistingCreditsCount', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'Job', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'Dependents', 'nullable': True, 'type': 'integer'}, {'metadata': {}, 'name': 'Telephone', 'nullable': True, 'type': 'string'}, {'metadata': {}, 'name': 'ForeignWorker', 'nullable': True, 'type': 'string'}, {'metadata': {'modeling_role': 'target'}, 'name': 'Risk', 'nullable': True, 'type': 'string'}], 'id': '1', 'type': 'struct'}, 'type': 'fs'}], 'type': 'mllib_3.3'}, 'metadata': {'created_at': '2024-08-07T06:34:23.751Z', 'description': 'German Credit Risk Model - Prod', 'id': '38f4cc7c-7231-4703-95bc-af7601819ddd', 'modified_at': '2024-08-07T06:34:31.788Z', 'name': 'German Credit Risk Model - Prod', 'owner': '1000331001', 'resource_key': '5ab9b73b-cbb2-45a4-9571-56ce4201c1c9', 'space_id': '00969846-29b6-446a-959e-587df21992e0'}, 'system': {'warnings': [{'message': 'Software specification spark-mllib_3.3 specified for the wml_model is deprecated and will be removed in the future. We recommend you use spark-mllib_3.4 instead. For details see Supported Frameworks https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=specifications-supported-deployment-frameworks'}]}}\n",
      "------------------------------------  -------------------------------  ------------------------  ---------  ----------  ----------------\n",
      "ID                                    NAME                             CREATED                   TYPE       SPEC_STATE  SPEC_REPLACEMENT\n",
      "38f4cc7c-7231-4703-95bc-af7601819ddd  German Credit Risk Model - Prod  2024-08-07T06:34:23.002Z  mllib_3.3  deprecated  spark-mllib_3.4\n",
      "------------------------------------  -------------------------------  ------------------------  ---------  ----------  ----------------\n",
      "Deploying model...\n",
      "\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "Synchronous deployment creation for uid: '38f4cc7c-7231-4703-95bc-af7601819ddd' started\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "\n",
      "initializing\n",
      "Note: Software specification spark-mllib_3.3 is deprecated. Use spark-mllib_3.4 software specification instead when saving a spark model. For details, see https://www.ibm.com/support/producthub/icpdata/docs/content/SSQNUZ_latest/wsj/wmls/wmls-deploy-python-types.html.\n",
      ".\n",
      "ready\n",
      "\n",
      "\n",
      "------------------------------------------------------------------------------------------------\n",
      "Successfully finished deployment creation, deployment_uid='2a85d08a-ea0b-4778-a76d-49099810f8dc'\n",
      "------------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "Model id: 38f4cc7c-7231-4703-95bc-af7601819ddd\n",
      "Deployment id: 2a85d08a-ea0b-4778-a76d-49099810f8dc\n",
      "Single record scoring result: \n",
      " fields: ['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker', 'CheckingStatus_IX', 'CreditHistory_IX', 'EmploymentDuration_IX', 'ExistingSavings_IX', 'ForeignWorker_IX', 'Housing_IX', 'InstallmentPlans_IX', 'Job_IX', 'LoanPurpose_IX', 'OthersOnLoan_IX', 'OwnsProperty_IX', 'Sex_IX', 'Telephone_IX', 'features', 'rawPrediction', 'probability', 'prediction', 'predictedLabel'] \n",
      " values:  ['no_checking', 13, 'credits_paid_to_date', 'car_new', 1343, '100_to_500', '1_to_4', 2, 'female', 'none', 3, 'savings_insurance', 46, 'none', 'own', 2, 'skilled', 1, 'none', 'yes', 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, [21, [1, 3, 5, 13, 14, 15, 16, 17, 18, 19, 20], [1.0, 1.0, 1.0, 13.0, 1343.0, 2.0, 3.0, 13.0, 46.0, 2.0, 1.0]], [0.37923501094948475, -0.37923501094948475], [0.6810214668555962, 0.31897853314440383], 0.0, 'No Risk']\n"
     ]
    }
   ],
   "source": [
    "import numpy \n",
    "numpy.version.version\n",
    "\n",
    "import pandas as pd\n",
    "import json\n",
    "\n",
    "from pyspark import SparkContext, SQLContext\n",
    "from pyspark.ml import Pipeline\n",
    "from pyspark.ml.classification import RandomForestClassifier,GBTClassifier\n",
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "from pyspark.ml.feature import StringIndexer, VectorAssembler, IndexToString\n",
    "from pyspark.sql.types import StructType, DoubleType, StringType, ArrayType\n",
    "\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark import SparkFiles\n",
    "\n",
    "spark = SparkSession.builder.getOrCreate()\n",
    "pd_data = pd.read_csv(\"german_credit_data_biased_training.csv\", sep=\",\", header=0)\n",
    "spark_df = spark.read.csv(path=\"german_credit_data_biased_training.csv\", sep=\",\", header=True, inferSchema=True)\n",
    "spark_df.head()\n",
    "\n",
    "(train_data, test_data) = spark_df.randomSplit([0.9, 0.1], 24)\n",
    "print(\"Number of records for training: \" + str(train_data.count()))\n",
    "print(\"Number of records for evaluation: \" + str(test_data.count()))\n",
    "\n",
    "si_CheckingStatus = StringIndexer(inputCol='CheckingStatus', outputCol='CheckingStatus_IX')\n",
    "si_CreditHistory = StringIndexer(inputCol='CreditHistory', outputCol='CreditHistory_IX')\n",
    "si_LoanPurpose = StringIndexer(inputCol='LoanPurpose', outputCol='LoanPurpose_IX')\n",
    "si_ExistingSavings = StringIndexer(inputCol='ExistingSavings', outputCol='ExistingSavings_IX')\n",
    "si_EmploymentDuration = StringIndexer(inputCol='EmploymentDuration', outputCol='EmploymentDuration_IX')\n",
    "si_Sex = StringIndexer(inputCol='Sex', outputCol='Sex_IX')\n",
    "si_OthersOnLoan = StringIndexer(inputCol='OthersOnLoan', outputCol='OthersOnLoan_IX')\n",
    "si_OwnsProperty = StringIndexer(inputCol='OwnsProperty', outputCol='OwnsProperty_IX')\n",
    "si_InstallmentPlans = StringIndexer(inputCol='InstallmentPlans', outputCol='InstallmentPlans_IX')\n",
    "si_Housing = StringIndexer(inputCol='Housing', outputCol='Housing_IX')\n",
    "si_Job = StringIndexer(inputCol='Job', outputCol='Job_IX')\n",
    "si_Telephone = StringIndexer(inputCol='Telephone', outputCol='Telephone_IX')\n",
    "si_ForeignWorker = StringIndexer(inputCol='ForeignWorker', outputCol='ForeignWorker_IX')\n",
    "si_Label = StringIndexer(inputCol=\"Risk\", outputCol=\"label\").fit(spark_df)\n",
    "label_converter = IndexToString(inputCol=\"prediction\", outputCol=\"predictedLabel\", labels=si_Label.labels)\n",
    "\n",
    "va_features = VectorAssembler(\n",
    "inputCols=[\"CheckingStatus_IX\", \"CreditHistory_IX\", \"LoanPurpose_IX\", \"ExistingSavings_IX\",\n",
    "           \"EmploymentDuration_IX\", \"Sex_IX\", \"OthersOnLoan_IX\", \"OwnsProperty_IX\", \"InstallmentPlans_IX\",\n",
    "           \"Housing_IX\", \"Job_IX\", \"Telephone_IX\", \"ForeignWorker_IX\", \"LoanDuration\", \"LoanAmount\",\n",
    "           \"InstallmentPercent\", \"CurrentResidenceDuration\", \"LoanDuration\", \"Age\", \"ExistingCreditsCount\",\n",
    "           \"Dependents\"], outputCol=\"features\")\n",
    "\n",
    "classifier=GBTClassifier(featuresCol=\"features\")\n",
    "\n",
    "pipeline = Pipeline(\n",
    "stages=[si_CheckingStatus, si_CreditHistory, si_EmploymentDuration, si_ExistingSavings, si_ForeignWorker,\n",
    "        si_Housing, si_InstallmentPlans, si_Job, si_LoanPurpose, si_OthersOnLoan,\n",
    "        si_OwnsProperty, si_Sex, si_Telephone, si_Label, va_features, classifier, label_converter])\n",
    "\n",
    "model = pipeline.fit(train_data)\n",
    "predictions = model.transform(test_data)\n",
    "evaluator = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\")\n",
    "auc = evaluator.evaluate(predictions)\n",
    "\n",
    "print(\"Accuracy = %g\" % auc)\n",
    "\n",
    "# Remove existing model and deployment\n",
    "MODEL_NAME=PROD_MODEL_NAME\n",
    "DEPLOYMENT_NAME=PROD_DEPLOYMENT_NAME\n",
    "\n",
    "deployment_details = wml_client.deployments.get_details()\n",
    "for deployment in deployment_details['resources']:\n",
    "    deployment_id = wml_client.deployments.get_id(deployment)\n",
    "    model_id = deployment['entity']['asset']['id']\n",
    "    if deployment['entity']['name'] == DEPLOYMENT_NAME:\n",
    "        print('Deleting deployment id', deployment_id)\n",
    "        wml_client.deployments.delete(deployment_id)\n",
    "        print('Deleting model id', model_id)\n",
    "        wml_client.repository.delete(model_id)\n",
    "wml_client.repository.list_models()\n",
    "\n",
    "# Save Model\n",
    "software_spec_uid = wml_client.software_specifications.get_id_by_name(\"spark-mllib_3.3\")\n",
    "model_props_rf = {\n",
    "    wml_client.repository.ModelMetaNames.NAME: MODEL_NAME,\n",
    "    wml_client.repository.ModelMetaNames.DESCRIPTION: MODEL_NAME,\n",
    "    wml_client.repository.ModelMetaNames.SOFTWARE_SPEC_UID: software_spec_uid,\n",
    "    wml_client.repository.ModelMetaNames.TYPE: 'mllib_3.3'\n",
    "}\n",
    "published_model_details = wml_client.repository.store_model(model=model, meta_props=model_props_rf, training_data=train_data, pipeline=pipeline)\n",
    "print(published_model_details)\n",
    "\n",
    "# List models in the repository\n",
    "wml_client.repository.list_models()\n",
    "\n",
    "# Get the model UID\n",
    "prod_model_uid = wml_client.repository.get_model_id(published_model_details)\n",
    "prod_model_uid\n",
    "\n",
    "\n",
    "# Deploy model\n",
    "wml_deployments = wml_client.deployments.get_details()\n",
    "prod_deployment_uid = None\n",
    "for deployment in wml_deployments['resources']:\n",
    "    if DEPLOYMENT_NAME == deployment['entity']['name']:\n",
    "        prod_deployment_uid = wml_client.deployments.get_id(deployment)\n",
    "        break\n",
    "\n",
    "if prod_deployment_uid is None:\n",
    "    print(\"Deploying model...\")\n",
    "    meta_props = {\n",
    "        wml_client.deployments.ConfigurationMetaNames.NAME: DEPLOYMENT_NAME,\n",
    "        wml_client.deployments.ConfigurationMetaNames.DESCRIPTION: DEPLOYMENT_NAME,\n",
    "        wml_client.deployments.ConfigurationMetaNames.ONLINE: {}\n",
    "    }\n",
    "    deployment = wml_client.deployments.create(artifact_uid=prod_model_uid, name=DEPLOYMENT_NAME, meta_props=meta_props)\n",
    "    prod_deployment_uid = wml_client.deployments.get_id(deployment)\n",
    "\n",
    "print(\"Model id: {}\".format(prod_model_uid))\n",
    "print(\"Deployment id: {}\".format(prod_deployment_uid))\n",
    "\n",
    "prod_deployment_uid=wml_client.deployments.get_id(deployment)\n",
    "prod_deployment_uid\n",
    "\n",
    "fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\n",
    "values = [\n",
    "  [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n",
    "  [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n",
    "]\n",
    "\n",
    "payload_scoring = {\"fields\": fields,\"values\": values}\n",
    "payload = {\n",
    "    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n",
    "}\n",
    "scoring_response = wml_client.deployments.score(prod_deployment_uid, payload)\n",
    "\n",
    "print('Single record scoring result:', '\\n fields:', scoring_response['predictions'][0]['fields'], '\\n values: ', scoring_response['predictions'][0]['values'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "id": "f6896d25dd824acb9799523b6ed6b651"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: Software specification spark-mllib_3.3 is deprecated. Use spark-mllib_3.4 software specification instead when saving a spark model. For details, see https://www.ibm.com/support/producthub/icpdata/docs/content/SSQNUZ_latest/wsj/wmls/wmls-deploy-python-types.html.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'entity': {'asset': {'id': '38f4cc7c-7231-4703-95bc-af7601819ddd'},\n",
       "  'custom': {},\n",
       "  'deployed_asset_type': 'model',\n",
       "  'description': 'German Credit Risk Model - Prod',\n",
       "  'hardware_spec': {'id': 'e7ed1d6c-2e89-42d7-aed5-863b972c1d2b',\n",
       "   'name': 'S',\n",
       "   'num_nodes': 1},\n",
       "  'name': 'German Credit Risk Model - Prod',\n",
       "  'online': {},\n",
       "  'space_id': '00969846-29b6-446a-959e-587df21992e0',\n",
       "  'status': {'inference': [{'url': 'https://cpd-cpd-instance.apps.wos415nfs2672.cp.fyre.ibm.com/ml/v4/deployments/2a85d08a-ea0b-4778-a76d-49099810f8dc/predictions'}],\n",
       "   'online_url': {'url': 'https://cpd-cpd-instance.apps.wos415nfs2672.cp.fyre.ibm.com/ml/v4/deployments/2a85d08a-ea0b-4778-a76d-49099810f8dc/predictions'},\n",
       "   'serving_urls': ['https://cpd-cpd-instance.apps.wos415nfs2672.cp.fyre.ibm.com/ml/v4/deployments/2a85d08a-ea0b-4778-a76d-49099810f8dc/predictions'],\n",
       "   'state': 'ready'}},\n",
       " 'metadata': {'created_at': '2024-08-07T06:34:37.217Z',\n",
       "  'description': 'German Credit Risk Model - Prod',\n",
       "  'id': '2a85d08a-ea0b-4778-a76d-49099810f8dc',\n",
       "  'modified_at': '2024-08-07T06:34:37.217Z',\n",
       "  'name': 'German Credit Risk Model - Prod',\n",
       "  'owner': '1000331001',\n",
       "  'space_id': '00969846-29b6-446a-959e-587df21992e0'},\n",
       " 'system': {'warnings': [{'id': 'Deprecated',\n",
       "    'message': 'Software specification spark-mllib_3.3 is deprecated. Use spark-mllib_3.4 software specification instead when saving a spark model. For details, see https://www.ibm.com/support/producthub/icpdata/docs/content/SSQNUZ_latest/wsj/wmls/wmls-deploy-python-types.html.'},\n",
       "   {'id': 'Deprecated',\n",
       "    'message': 'online_url is deprecated and will be removed in a future release. Use serving_urls instead.'}]}}"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wml_client.deployments.get_details(prod_deployment_uid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "15003742-8f20-4747-82c9-20fdd6279d86"
   },
   "source": [
    "## Bind WML machine learning instance as Prod\n",
    "\n",
    "Watson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model. If a binding with name \"WML Prod\" already exists, this code will delete that binding a create a new one.\n",
    "\n",
    "**Note**: Binding with name `WML Prod` is assumed to be only created by this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "id": "164bd39becbd45d286cbc17a6b999ca0"
   },
   "outputs": [],
   "source": [
    "SERVICE_PROVIDER_NAME = 'WML Prod'\n",
    "SERVICE_PROVIDER_DESCRIPTION = 'Prod service provider created by MRM tutorial'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "id": "28366db5-b7b4-4cc8-aae1-1205bfaa15f1"
   },
   "outputs": [],
   "source": [
    "service_providers = wos_client.service_providers.list().result.service_providers\n",
    "for service_provider in service_providers:\n",
    "    service_instance_name = service_provider.entity.name\n",
    "    if service_instance_name == SERVICE_PROVIDER_NAME:\n",
    "        service_provider_id = service_provider.metadata.id\n",
    "        wos_client.service_providers.delete(service_provider_id)\n",
    "        print(\"Deleted existing service_provider for WML instance: {}\".format(service_provider_id))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "id": "d011bc288a184df987ffb5ecc782adba"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "=================================================================================\n",
      "\n",
      " Waiting for end of adding service provider f4931b54-2103-40d9-b564-9ba3b1982fdb \n",
      "\n",
      "=================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "-----------------------------------------------\n",
      " Successfully finished adding service provider \n",
      "-----------------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'f4931b54-2103-40d9-b564-9ba3b1982fdb'"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "added_service_provider_result = wos_client.service_providers.add(\n",
    "        name=SERVICE_PROVIDER_NAME,\n",
    "        description=SERVICE_PROVIDER_DESCRIPTION,\n",
    "        service_type=ServiceTypes.WATSON_MACHINE_LEARNING,\n",
    "        deployment_space_id = space_id,\n",
    "        operational_space_id = \"production\",\n",
    "        credentials=WMLCredentialsCP4D(),\n",
    "        background_mode=False\n",
    "    ).result\n",
    "service_provider_id = added_service_provider_result.metadata.id\n",
    "service_provider_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "id": "d351eda7-2aed-48b3-95ce-2763e6c71688"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'metadata': {'id': 'f4931b54-2103-40d9-b564-9ba3b1982fdb',\n",
       "  'crn': 'crn:v1:bluemix:public:aiopenscale:us-south:a/na:00000000-0000-0000-0000-000000000000:service_provider:f4931b54-2103-40d9-b564-9ba3b1982fdb',\n",
       "  'url': '/v2/service_providers/f4931b54-2103-40d9-b564-9ba3b1982fdb',\n",
       "  'created_at': '2024-08-07T06:34:58.909000Z',\n",
       "  'created_by': 'cpadmin'},\n",
       " 'entity': {'name': 'WML Prod',\n",
       "  'service_type': 'watson_machine_learning',\n",
       "  'instance_id': '99999999-9999-9999-9999-999999999999',\n",
       "  'credentials': {'secret_id': '09136882-f7e9-4645-8777-12af8449419b'},\n",
       "  'operational_space_id': 'production',\n",
       "  'deployment_space_id': '00969846-29b6-446a-959e-587df21992e0',\n",
       "  'status': {'state': 'active'}}}"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "added_service_provider_result.to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "37ec5e99-a8d0-43c6-8c3d-7570023a71cf"
   },
   "source": [
    "## Remove existing prod subscription\n",
    "\n",
    "This code removes previous subscription that matches the name `German Credit Risk Model - Prod` as it is expected this subscription is created only via this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "id": "c77139ad-8952-42fe-bfcc-c91bc76438bf"
   },
   "outputs": [],
   "source": [
    "subscriptions = wos_client.subscriptions.list().result.subscriptions\n",
    "for subscription in subscriptions:\n",
    "    sub_name = subscription.entity.asset.name\n",
    "    if sub_name == PROD_MODEL_NAME:\n",
    "        wos_client.subscriptions.delete(subscription.metadata.id)\n",
    "        print('Deleted existing subscription for', sub_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1be0d17e-11eb-4a06-b376-a1a03827ea7a"
   },
   "source": [
    "# Import configuration settings from pre-prod model\n",
    "\n",
    "With MRM we provide a important feature that lets you copy the configuration settings of your pre-production subscription to the production subscription. To try this out\n",
    "\n",
    "1. Navigate to Model Monitors view in Insights dashboard of OpenScale\n",
    "2. Click on the Add to dashboard\n",
    "3. Select the production model deployment from WML production machine learning provider and click on Configure\n",
    "4. In Selections saved dialog, click on Configure monitors\n",
    "5. Click on Import settings\n",
    "6. In the Import configuration settings dialog, choose the `German Credit Risk Model - PreProd` as the subscription from which you want to import the settings and click Configure\n",
    "7. In the Replace existing settings? dialog, click on Import\n",
    "\n",
    "All the configuration settings are now copied into the production subscription\n",
    "\n",
    "\n",
    "<b>Note: The next set of cells should be executed only after finishing the import settings from the OpenScale dashboard</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bf9715aa-cbca-4e41-9078-5d3a9525ebb7"
   },
   "source": [
    "## Score the production model so that we can trigger monitors\n",
    "\n",
    "Now that the production subscription is configured by copying the configuration, there would be schedules created for each of the monitors to run on a scheduled basis. \n",
    "Quality, Fairness and Mrm will run hourly. Drift will run once in three hours.\n",
    "\n",
    "For this demo purpose, we will trigger the monitors on-demand so that we can see the model summary dashboard without having to wait the entire hour. \n",
    "To do that lets first push some records in the Payload Logging table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "id": "041e607a-563a-47df-9316-c8898c4d56bd"
   },
   "outputs": [],
   "source": [
    "df = pd_data.sample(n=400)\n",
    "df = df.drop(['Risk'], axis=1)\n",
    "fields = df.columns.tolist()\n",
    "values = df.values.tolist()\n",
    "\n",
    "payload_scoring = {\"fields\": fields,\"values\": values}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "id": "e22a78d1-4d47-4327-b862-f8c113dc561c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Single record scoring result: \n",
      " fields: ['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker', 'CheckingStatus_IX', 'CreditHistory_IX', 'EmploymentDuration_IX', 'ExistingSavings_IX', 'ForeignWorker_IX', 'Housing_IX', 'InstallmentPlans_IX', 'Job_IX', 'LoanPurpose_IX', 'OthersOnLoan_IX', 'OwnsProperty_IX', 'Sex_IX', 'Telephone_IX', 'features', 'rawPrediction', 'probability', 'prediction', 'predictedLabel'] \n",
      " values:  ['less_0', 11, 'all_credits_paid_back', 'car_new', 250, 'less_100', 'less_1', 1, 'female', 'none', 2, 'savings_insurance', 25, 'stores', 'rent', 1, 'skilled', 1, 'none', 'yes', 1.0, 3.0, 3.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, [1.0, 3.0, 0.0, 0.0, 3.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 11.0, 250.0, 1.0, 2.0, 11.0, 25.0, 1.0, 1.0], [1.3798199484285352, -1.3798199484285352], [0.9404554718138881, 0.05954452818611189], 0.0, 'No Risk']\n"
     ]
    }
   ],
   "source": [
    "payload = {\n",
    "    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]\n",
    "}\n",
    "scoring_response = wml_client.deployments.score(prod_deployment_uid, payload)\n",
    "\n",
    "print('Single record scoring result:', '\\n fields:', scoring_response['predictions'][0]['fields'], '\\n values: ', scoring_response['predictions'][0]['values'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "id": "75cd2a27-8dc7-4f2b-9d99-6881c810b876"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "61003e50-20d6-4ce8-a379-013a03c44229\n"
     ]
    }
   ],
   "source": [
    "subscriptions = wos_client.subscriptions.list().result.subscriptions\n",
    "for subscription in subscriptions:\n",
    "    sub_name = subscription.entity.asset.name\n",
    "    if sub_name == PROD_MODEL_NAME:\n",
    "        prod_subscription = subscription\n",
    "prod_subscription_id = pre_prod_subscription.metadata.id\n",
    "print(prod_subscription_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "id": "f793e126be92449096066d686c5746b6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'metadata': {'id': '61003e50-20d6-4ce8-a379-013a03c44229',\n",
       "  'crn': 'crn:v1:bluemix:public:aiopenscale:us-south:a/na:00000000-0000-0000-0000-000000000000:subscription:61003e50-20d6-4ce8-a379-013a03c44229',\n",
       "  'url': '/v2/subscriptions/61003e50-20d6-4ce8-a379-013a03c44229',\n",
       "  'created_at': '2024-08-07T06:07:24.153000Z',\n",
       "  'created_by': 'cpadmin',\n",
       "  'modified_at': '2024-08-07T06:07:26.863000Z',\n",
       "  'modified_by': 'cpadmin'},\n",
       " 'entity': {'data_mart_id': '00000000-0000-0000-0000-000000000000',\n",
       "  'service_provider_id': '81655d73-6ea3-44ae-9876-86df1fa6390d',\n",
       "  'asset': {'asset_id': 'f1b4fe35-1a75-4dca-a9b3-bc5dd7ed43db',\n",
       "   'url': 'https://internal-nginx-svc:12443/ml/v4/models/f1b4fe35-1a75-4dca-a9b3-bc5dd7ed43db?space_id=c3905865-4e73-43d1-a18f-24cf3a8b9163&version=2020-06-12',\n",
       "   'name': 'German Credit Risk Model - PreProd',\n",
       "   'asset_type': 'model',\n",
       "   'problem_type': 'binary',\n",
       "   'model_type': 'mllib_3.3',\n",
       "   'input_data_type': 'structured'},\n",
       "  'asset_properties': {'training_data_reference': {'secret_id': '785e2291-4ec7-4926-b3b8-900bc6c167ab'},\n",
       "   'training_data_schema': {'type': 'struct',\n",
       "    'fields': [{'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'CheckingStatus',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'LoanDuration',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'CreditHistory',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'LoanPurpose',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'LoanAmount',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'ExistingSavings',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'EmploymentDuration',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'InstallmentPercent',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'Sex',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'OthersOnLoan',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'CurrentResidenceDuration',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'OwnsProperty',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'Age',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'InstallmentPlans',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'Housing',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'ExistingCreditsCount',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'Job',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'Dependents',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'Telephone',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'ForeignWorker',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'modeling_role': 'target'},\n",
       "      'name': 'Risk',\n",
       "      'nullable': True,\n",
       "      'type': 'string'}]},\n",
       "   'input_data_schema': {'type': 'struct',\n",
       "    'fields': [{'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'CheckingStatus',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'LoanDuration',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'CreditHistory',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'LoanPurpose',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'LoanAmount',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'ExistingSavings',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'EmploymentDuration',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'InstallmentPercent',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'Sex',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'OthersOnLoan',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'CurrentResidenceDuration',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'OwnsProperty',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'Age',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'InstallmentPlans',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'Housing',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'ExistingCreditsCount',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'Job',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'Dependents',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'Telephone',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'ForeignWorker',\n",
       "      'nullable': True,\n",
       "      'type': 'string'}]},\n",
       "   'output_data_schema': {'type': 'struct',\n",
       "    'fields': [{'metadata': {'columnInfo': {'columnLength': 128},\n",
       "       'modeling_role': 'record-id',\n",
       "       'primary_key': True},\n",
       "      'name': 'scoring_id',\n",
       "      'nullable': False,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'record-timestamp'},\n",
       "      'name': 'scoring_timestamp',\n",
       "      'nullable': False,\n",
       "      'type': 'timestamp'},\n",
       "     {'metadata': {},\n",
       "      'name': 'deployment_id',\n",
       "      'nullable': False,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {},\n",
       "      'name': 'asset_revision',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'CheckingStatus',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'LoanDuration',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'CreditHistory',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'LoanPurpose',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'LoanAmount',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'ExistingSavings',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'EmploymentDuration',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'InstallmentPercent',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'Sex',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'OthersOnLoan',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'CurrentResidenceDuration',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'OwnsProperty',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'Age',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'InstallmentPlans',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'Housing',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'ExistingCreditsCount',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'Job',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'feature'},\n",
       "      'name': 'Dependents',\n",
       "      'nullable': True,\n",
       "      'type': 'integer'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'Telephone',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'measure': 'discrete',\n",
       "       'modeling_role': 'feature'},\n",
       "      'name': 'ForeignWorker',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {},\n",
       "      'name': 'CheckingStatus_IX',\n",
       "      'nullable': True,\n",
       "      'type': 'double'},\n",
       "     {'metadata': {},\n",
       "      'name': 'CreditHistory_IX',\n",
       "      'nullable': True,\n",
       "      'type': 'double'},\n",
       "     {'metadata': {},\n",
       "      'name': 'EmploymentDuration_IX',\n",
       "      'nullable': True,\n",
       "      'type': 'double'},\n",
       "     {'metadata': {},\n",
       "      'name': 'ExistingSavings_IX',\n",
       "      'nullable': True,\n",
       "      'type': 'double'},\n",
       "     {'metadata': {},\n",
       "      'name': 'ForeignWorker_IX',\n",
       "      'nullable': True,\n",
       "      'type': 'double'},\n",
       "     {'metadata': {},\n",
       "      'name': 'Housing_IX',\n",
       "      'nullable': True,\n",
       "      'type': 'double'},\n",
       "     {'metadata': {},\n",
       "      'name': 'InstallmentPlans_IX',\n",
       "      'nullable': True,\n",
       "      'type': 'double'},\n",
       "     {'metadata': {}, 'name': 'Job_IX', 'nullable': True, 'type': 'double'},\n",
       "     {'metadata': {},\n",
       "      'name': 'LoanPurpose_IX',\n",
       "      'nullable': True,\n",
       "      'type': 'double'},\n",
       "     {'metadata': {},\n",
       "      'name': 'OthersOnLoan_IX',\n",
       "      'nullable': True,\n",
       "      'type': 'double'},\n",
       "     {'metadata': {},\n",
       "      'name': 'OwnsProperty_IX',\n",
       "      'nullable': True,\n",
       "      'type': 'double'},\n",
       "     {'metadata': {}, 'name': 'Sex_IX', 'nullable': True, 'type': 'double'},\n",
       "     {'metadata': {},\n",
       "      'name': 'Telephone_IX',\n",
       "      'nullable': True,\n",
       "      'type': 'double'},\n",
       "     {'metadata': {},\n",
       "      'name': 'features',\n",
       "      'nullable': True,\n",
       "      'type': {'containsNull': True,\n",
       "       'elementType': 'double',\n",
       "       'type': 'array'}},\n",
       "     {'metadata': {},\n",
       "      'name': 'rawPrediction',\n",
       "      'nullable': True,\n",
       "      'type': {'containsNull': True,\n",
       "       'elementType': 'double',\n",
       "       'type': 'array'}},\n",
       "     {'metadata': {'modeling_role': 'probability'},\n",
       "      'name': 'probability',\n",
       "      'nullable': True,\n",
       "      'type': {'containsNull': True,\n",
       "       'elementType': 'double',\n",
       "       'type': 'array'}},\n",
       "     {'metadata': {},\n",
       "      'name': 'prediction',\n",
       "      'nullable': True,\n",
       "      'type': 'double'},\n",
       "     {'metadata': {'columnInfo': {'columnLength': 64},\n",
       "       'modeling_role': 'prediction'},\n",
       "      'name': 'predictedLabel',\n",
       "      'nullable': True,\n",
       "      'type': 'string'},\n",
       "     {'metadata': {'modeling_role': 'prediction-probability'},\n",
       "      'name': 'prediction_probability',\n",
       "      'nullable': True,\n",
       "      'type': 'double'}]},\n",
       "   'label_column': 'Risk',\n",
       "   'prediction_field': 'predictedLabel',\n",
       "   'feature_fields': ['CheckingStatus',\n",
       "    'LoanDuration',\n",
       "    'CreditHistory',\n",
       "    'LoanPurpose',\n",
       "    'LoanAmount',\n",
       "    'ExistingSavings',\n",
       "    'EmploymentDuration',\n",
       "    'InstallmentPercent',\n",
       "    'Sex',\n",
       "    'OthersOnLoan',\n",
       "    'CurrentResidenceDuration',\n",
       "    'OwnsProperty',\n",
       "    'Age',\n",
       "    'InstallmentPlans',\n",
       "    'Housing',\n",
       "    'ExistingCreditsCount',\n",
       "    'Job',\n",
       "    'Dependents',\n",
       "    'Telephone',\n",
       "    'ForeignWorker'],\n",
       "   'categorical_fields': ['CheckingStatus',\n",
       "    'CreditHistory',\n",
       "    'LoanPurpose',\n",
       "    'ExistingSavings',\n",
       "    'EmploymentDuration',\n",
       "    'Sex',\n",
       "    'OthersOnLoan',\n",
       "    'OwnsProperty',\n",
       "    'InstallmentPlans',\n",
       "    'Housing',\n",
       "    'Job',\n",
       "    'Telephone',\n",
       "    'ForeignWorker'],\n",
       "   'probability_fields': ['probability']},\n",
       "  'deployment': {'deployment_id': '42b13344-baa1-45a0-89ca-ef662387f691',\n",
       "   'url': 'https://cpd-cpd-instance.apps.wos415nfs2672.cp.fyre.ibm.com/ml/v4/deployments/42b13344-baa1-45a0-89ca-ef662387f691/predictions',\n",
       "   'name': 'German Credit Risk Model - PreProd',\n",
       "   'deployment_type': 'online'},\n",
       "  'status': {'state': 'active'}}}"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = wos_client.subscriptions.get(prod_subscription_id).result.to_dict()\n",
    "res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "id": "04f0c84a-6e72-474a-8e0b-ce4c658809a1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Payload data set id:  0da980a0-4be1-4bd8-85b7-8cf5b694d5a1\n",
      "Number of records in the payload logging table: 408\n"
     ]
    }
   ],
   "source": [
    "time.sleep(10)\n",
    "prod_pl_data_set_id = None\n",
    "prod_pl_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, \n",
    "                                                target_target_id=prod_subscription_id, \n",
    "                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id\n",
    "if prod_pl_data_set_id is None:\n",
    "    print(\"Payload data set not found. Please check subscription status.\")\n",
    "else:\n",
    "    print(\"Payload data set id: \", prod_pl_data_set_id)\n",
    "    pl_records_count = wos_client.data_sets.get_records_count(prod_pl_data_set_id)\n",
    "    print(\"Number of records in the payload logging table: {}\".format(pl_records_count))\n",
    "    if pl_records_count == 0:\n",
    "        print(\"Automatic payload logging did not happen, performing explicit payload logging.\")\n",
    "        wos_client.data_sets.store_records(data_set_id=prod_pl_data_set_id, request_body=[PayloadRecord(\n",
    "                       scoring_id='mrm_'+str(uuid.uuid4()),\n",
    "                       request=payload_scoring,\n",
    "                       response=scoring_response['predictions'][0]\n",
    "                   )])\n",
    "        time.sleep(5)\n",
    "        pl_records_count = wos_client.data_sets.get_records_count(prod_pl_data_set_id)\n",
    "        print(\"Number of records in the payload logging table: {}\".format(pl_records_count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bbb399a4-c647-4e78-8181-934b6646931c"
   },
   "source": [
    "## Fetch all monitor instances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "id": "0852a68b-67b3-4d14-b7b1-c06192d09923"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Quality monitor instance id - 85319c11-608f-4625-b38a-1b51c661c4fc\n",
      "Drift monitor instance id - b43d2e79-e244-44c2-9dc7-c5a7bae56a2e\n",
      "Drift v2 monitor instance id - d6b3f5bc-82f5-4042-b6b2-3ac1aab9eb07\n",
      "Fairness monitor instance id - 5bd1947b-f9df-42ba-82d2-ebafcbe5eb42\n",
      "MRM monitor instance id - 23b1f186-f599-45fb-830c-2810c18bad42\n"
     ]
    }
   ],
   "source": [
    "monitor_instances = wos_client.monitor_instances.list(target_target_id=prod_subscription_id).result.monitor_instances\n",
    "for monitor_instance in monitor_instances:\n",
    "    monitor_def_id=monitor_instance.entity.monitor_definition_id\n",
    "    if monitor_def_id == wos_client.monitor_definitions.MONITORS.QUALITY.ID:\n",
    "        quality_monitor_instance_id = monitor_instance.metadata.id\n",
    "    if monitor_def_id == wos_client.monitor_definitions.MONITORS.DRIFT.ID:\n",
    "        drift_monitor_instance_id = monitor_instance.metadata.id\n",
    "    if monitor_def_id == wos_client.monitor_definitions.MONITORS.DRIFT_V2.ID:\n",
    "        drift_v2_monitor_instance_id = monitor_instance.metadata.id\n",
    "    if monitor_def_id == wos_client.monitor_definitions.MONITORS.FAIRNESS.ID:\n",
    "        fairness_monitor_instance_id = monitor_instance.metadata.id\n",
    "    if monitor_def_id == 'mrm':\n",
    "        mrm_monitor_instance_id = monitor_instance.metadata.id\n",
    "                    \n",
    "print(\"Quality monitor instance id - {0}\".format(quality_monitor_instance_id))\n",
    "print(\"Drift monitor instance id - {0}\".format(drift_monitor_instance_id))\n",
    "print(\"Drift v2 monitor instance id - {0}\".format(drift_v2_monitor_instance_id))\n",
    "print(\"Fairness monitor instance id - {0}\".format(fairness_monitor_instance_id))\n",
    "print(\"MRM monitor instance id - {0}\".format(mrm_monitor_instance_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3ea2b1bc-d181-4ec1-95ea-37e64d3e586f"
   },
   "source": [
    "## Run on-demand Quality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f6c54f32bc9e44899f84e36b5f0b5ebd"
   },
   "source": [
    "### Feedback logging\n",
    "\n",
    "The code below downloads and stores enough feedback data to meet the minimum threshold so that OpenScale can calculate a new accuracy measurement. It then kicks off the accuracy monitor. The monitors run hourly, or can be initiated via the Python API, the REST API, or the graphical user interface.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "id": "07861f904230444a85225b0814103bfb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "97d9925d-4386-4384-a93c-769b14ffa003\n"
     ]
    }
   ],
   "source": [
    "feedback_dataset_id = None\n",
    "feedback_dataset = wos_client.data_sets.list(type=DataSetTypes.FEEDBACK, \n",
    "                                                target_target_id=prod_subscription_id, \n",
    "                                                target_target_type=TargetTypes.SUBSCRIPTION).result\n",
    "feedback_dataset_id = feedback_dataset.data_sets[0].metadata.id\n",
    "if feedback_dataset_id is None:\n",
    "    print(\"Feedback data set not found. Please check quality monitor status.\")\n",
    "print(feedback_dataset_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "id": "394b4292b27d42a58fb27034b6170bdd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm: additional_feedback_data_v2.json: No such file or directory\n",
      "--2024-08-07 12:05:51--  https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/additional_feedback_data_v2.json\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.109.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 50890 (50K) [text/plain]\n",
      "Saving to: ‘additional_feedback_data_v2.json’\n",
      "\n",
      "additional_feedback 100%[===================>]  49.70K  --.-KB/s    in 0.01s   \n",
      "\n",
      "2024-08-07 12:05:52 (3.81 MB/s) - ‘additional_feedback_data_v2.json’ saved [50890/50890]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm additional_feedback_data_v2.json\n",
    "!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/additional_feedback_data_v2.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "id": "077e9013ba414ad8b4c15bfa3413cb22"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "==========================================================================================\n",
      "\n",
      " Waiting for end of storing records with request id: 7b7aadac-f7dd-4abc-bc55-d6f58e5aab14 \n",
      "\n",
      "==========================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Successfully finished storing records \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<ibm_watson_openscale.base_classes.watson_open_scale_v2.Status at 0x148159f00>"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('additional_feedback_data_v2.json') as feedback_file:\n",
    "    additional_feedback_data = json.load(feedback_file)\n",
    "wos_client.data_sets.store_records(feedback_dataset_id, request_body=additional_feedback_data, background_mode=False).result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "id": "10b0d41eaba14fc883405ab8e0393a5f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of records in the feedback table:  196\n"
     ]
    }
   ],
   "source": [
    "time.sleep(5)\n",
    "feedback_records_count = wos_client.data_sets.get_records_count(data_set_id=feedback_dataset_id)\n",
    "print('Number of records in the feedback table: ', feedback_records_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "id": "daa085fd-1821-485b-8681-d0751192d42d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "========================================================================\n",
      "\n",
      " Waiting for end of monitoring run 5b133df1-f070-40c1-9e1e-331f2853b365 \n",
      "\n",
      "========================================================================\n",
      "\n",
      "\n",
      "\n",
      "running\n",
      "finished\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "if quality_monitor_instance_id is not None:\n",
    "    run_details = wos_client.monitor_instances.run(monitor_instance_id=quality_monitor_instance_id, background_mode=False).result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "id": "0e25e9071bae47b6b8d96c0c3a74055c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>85319c11-608f-4625-b38a-1b51c661c4fc Monitor Runs Metrics from: 2024-07-31 12:06:57.799591  till: 2024-08-07 12:06:57.799596</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower_limit</th><th style='border: 1px solid #dddddd'>upper_limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>monitor_instance_id</th><th style='border: 1px solid #dddddd'>run_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>target_id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2024-08-07 06:36:46.020000+00:00</td><td style='border: 1px solid #dddddd'>true_positive_rate</td><td style='border: 1px solid #dddddd'>3c2ce1cf-c376-43a2-aa74-512cdba4371d</td><td style='border: 1px solid #dddddd'>0.36363636363636365</td><td style='border: 1px solid #dddddd'>0.8</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>85319c11-608f-4625-b38a-1b51c661c4fc</td><td style='border: 1px solid #dddddd'>5b133df1-f070-40c1-9e1e-331f2853b365</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:36:46.020000+00:00</td><td style='border: 1px solid #dddddd'>area_under_roc</td><td style='border: 1px solid #dddddd'>3c2ce1cf-c376-43a2-aa74-512cdba4371d</td><td style='border: 1px solid #dddddd'>0.6510489510489511</td><td style='border: 1px solid #dddddd'>0.8</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>85319c11-608f-4625-b38a-1b51c661c4fc</td><td style='border: 1px solid #dddddd'>5b133df1-f070-40c1-9e1e-331f2853b365</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:36:46.020000+00:00</td><td style='border: 1px solid #dddddd'>precision</td><td style='border: 1px solid #dddddd'>3c2ce1cf-c376-43a2-aa74-512cdba4371d</td><td style='border: 1px solid #dddddd'>0.75</td><td style='border: 1px solid #dddddd'>0.8</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>85319c11-608f-4625-b38a-1b51c661c4fc</td><td style='border: 1px solid #dddddd'>5b133df1-f070-40c1-9e1e-331f2853b365</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:36:46.020000+00:00</td><td style='border: 1px solid #dddddd'>matthews_correlation_coefficient</td><td style='border: 1px solid #dddddd'>3c2ce1cf-c376-43a2-aa74-512cdba4371d</td><td style='border: 1px solid #dddddd'>0.38627307784817894</td><td style='border: 1px solid #dddddd'>0.8</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>85319c11-608f-4625-b38a-1b51c661c4fc</td><td style='border: 1px solid #dddddd'>5b133df1-f070-40c1-9e1e-331f2853b365</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:36:46.020000+00:00</td><td style='border: 1px solid #dddddd'>f1_measure</td><td style='border: 1px solid #dddddd'>3c2ce1cf-c376-43a2-aa74-512cdba4371d</td><td style='border: 1px solid #dddddd'>0.48979591836734687</td><td style='border: 1px solid #dddddd'>0.8</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>85319c11-608f-4625-b38a-1b51c661c4fc</td><td style='border: 1px solid #dddddd'>5b133df1-f070-40c1-9e1e-331f2853b365</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:36:46.020000+00:00</td><td style='border: 1px solid #dddddd'>accuracy</td><td style='border: 1px solid #dddddd'>3c2ce1cf-c376-43a2-aa74-512cdba4371d</td><td style='border: 1px solid #dddddd'>0.7448979591836735</td><td style='border: 1px solid #dddddd'>0.8</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>85319c11-608f-4625-b38a-1b51c661c4fc</td><td style='border: 1px solid #dddddd'>5b133df1-f070-40c1-9e1e-331f2853b365</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:36:46.020000+00:00</td><td style='border: 1px solid #dddddd'>label_skew</td><td style='border: 1px solid #dddddd'>3c2ce1cf-c376-43a2-aa74-512cdba4371d</td><td style='border: 1px solid #dddddd'>0.690933627340049</td><td style='border: 1px solid #dddddd'>-0.5</td><td style='border: 1px solid #dddddd'>0.5</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>85319c11-608f-4625-b38a-1b51c661c4fc</td><td style='border: 1px solid #dddddd'>5b133df1-f070-40c1-9e1e-331f2853b365</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:36:46.020000+00:00</td><td style='border: 1px solid #dddddd'>gini_coefficient</td><td style='border: 1px solid #dddddd'>3c2ce1cf-c376-43a2-aa74-512cdba4371d</td><td style='border: 1px solid #dddddd'>0.3020979020979022</td><td style='border: 1px solid #dddddd'>0.8</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>85319c11-608f-4625-b38a-1b51c661c4fc</td><td style='border: 1px solid #dddddd'>5b133df1-f070-40c1-9e1e-331f2853b365</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:36:46.020000+00:00</td><td style='border: 1px solid #dddddd'>log_loss</td><td style='border: 1px solid #dddddd'>3c2ce1cf-c376-43a2-aa74-512cdba4371d</td><td style='border: 1px solid #dddddd'>0.43609695877817856</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.8</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>85319c11-608f-4625-b38a-1b51c661c4fc</td><td style='border: 1px solid #dddddd'>5b133df1-f070-40c1-9e1e-331f2853b365</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:36:46.020000+00:00</td><td style='border: 1px solid #dddddd'>false_positive_rate</td><td style='border: 1px solid #dddddd'>3c2ce1cf-c376-43a2-aa74-512cdba4371d</td><td style='border: 1px solid #dddddd'>0.06153846153846154</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.8</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>85319c11-608f-4625-b38a-1b51c661c4fc</td><td style='border: 1px solid #dddddd'>5b133df1-f070-40c1-9e1e-331f2853b365</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: First 10 records were displayed.\n"
     ]
    }
   ],
   "source": [
    "wos_client.monitor_instances.show_metrics(monitor_instance_id=quality_monitor_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "df8145d0-f3b9-4fbb-8605-7608c84731be"
   },
   "source": [
    "## Run on-demand Drift"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "id": "1d167f82-54e9-491e-959a-da4689fb8dff"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "========================================================================\n",
      "\n",
      " Waiting for end of monitoring run 575020ec-722a-4b1f-bc89-c3b3f6a344b7 \n",
      "\n",
      "========================================================================\n",
      "\n",
      "\n",
      "\n",
      "running.\n",
      "finished\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "if drift_monitor_instance_id is not None:\n",
    "    run_details = wos_client.monitor_instances.run(monitor_instance_id=drift_monitor_instance_id, background_mode=False).result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "id": "b9ab6682-e76f-4507-ae4d-0c420c352ce0"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>b43d2e79-e244-44c2-9dc7-c5a7bae56a2e Monitor Runs Metrics from: 2024-07-31 12:09:21.525729  till: 2024-08-07 12:09:21.525735</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower_limit</th><th style='border: 1px solid #dddddd'>upper_limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>monitor_instance_id</th><th style='border: 1px solid #dddddd'>run_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>target_id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2024-08-07 06:39:05.836039+00:00</td><td style='border: 1px solid #dddddd'>data_drift_magnitude</td><td style='border: 1px solid #dddddd'>cfa28d65-75f5-4371-8c6c-4a7251cf0098</td><td style='border: 1px solid #dddddd'>0.07</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.1</td><td style='border: 1px solid #dddddd'>[]</td><td style='border: 1px solid #dddddd'>drift</td><td style='border: 1px solid #dddddd'>b43d2e79-e244-44c2-9dc7-c5a7bae56a2e</td><td style='border: 1px solid #dddddd'>575020ec-722a-4b1f-bc89-c3b3f6a344b7</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wos_client.monitor_instances.show_metrics(monitor_instance_id=drift_monitor_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run on-demand Drift V2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.code.notebook.stdout": [
       "\n",
       "\n",
       "========================================================================\n",
       "\n",
       " Waiting for end of monitoring run fe74deb9-2d5e-4cb5-ad35-6cfaec49543a \n",
       "\n",
       "========================================================================\n",
       "\n",
       "\n",
       "\n",
       "running.............\n",
       "finished\n",
       "\n",
       "---------------------------\n",
       " Successfully finished run \n",
       "---------------------------\n",
       "\n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if drift_v2_monitor_instance_id is not None:\n",
    "    run_details = wos_client.monitor_instances.run(monitor_instance_id=drift_v2_monitor_instance_id, background_mode=False).result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>ddf7db84-b9a6-4436-93c4-e386a5adb8d7 Monitor Runs Metrics from: 2024-07-30 18:38:28.557913  till: 2024-08-06 18:38:28.557941</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower_limit</th><th style='border: 1px solid #dddddd'>upper_limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>monitor_instance_id</th><th style='border: 1px solid #dddddd'>run_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>target_id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2024-08-06 13:03:30.703769+00:00</td><td style='border: 1px solid #dddddd'>records_processed</td><td style='border: 1px solid #dddddd'>79b5fec3-52e9-4ccb-8d28-a48adcd894e2</td><td style='border: 1px solid #dddddd'>408.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['algorithm_used:total_variation', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for No Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>ddf7db84-b9a6-4436-93c4-e386a5adb8d7</td><td style='border: 1px solid #dddddd'>ed70ede4-5f54-491c-86c1-9845d7530447</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>e4098249-ffec-436c-b76d-0b2ffeec5bf4</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-06 13:03:30.703769+00:00</td><td style='border: 1px solid #dddddd'>confidence_drift_score</td><td style='border: 1px solid #dddddd'>79b5fec3-52e9-4ccb-8d28-a48adcd894e2</td><td style='border: 1px solid #dddddd'>0.1534</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.05</td><td style='border: 1px solid #dddddd'>['algorithm_used:total_variation', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for No Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>ddf7db84-b9a6-4436-93c4-e386a5adb8d7</td><td style='border: 1px solid #dddddd'>ed70ede4-5f54-491c-86c1-9845d7530447</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>e4098249-ffec-436c-b76d-0b2ffeec5bf4</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-06 13:03:30.703769+00:00</td><td style='border: 1px solid #dddddd'>records_processed</td><td style='border: 1px solid #dddddd'>79b5fec3-52e9-4ccb-8d28-a48adcd894e2</td><td style='border: 1px solid #dddddd'>408.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['algorithm_used:overlap_coefficient', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for No Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>ddf7db84-b9a6-4436-93c4-e386a5adb8d7</td><td style='border: 1px solid #dddddd'>ed70ede4-5f54-491c-86c1-9845d7530447</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>e4098249-ffec-436c-b76d-0b2ffeec5bf4</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-06 13:03:30.703769+00:00</td><td style='border: 1px solid #dddddd'>confidence_drift_score</td><td style='border: 1px solid #dddddd'>79b5fec3-52e9-4ccb-8d28-a48adcd894e2</td><td style='border: 1px solid #dddddd'>0.1552</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.05</td><td style='border: 1px solid #dddddd'>['algorithm_used:overlap_coefficient', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for No Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>ddf7db84-b9a6-4436-93c4-e386a5adb8d7</td><td style='border: 1px solid #dddddd'>ed70ede4-5f54-491c-86c1-9845d7530447</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>e4098249-ffec-436c-b76d-0b2ffeec5bf4</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-06 13:03:38.611350+00:00</td><td style='border: 1px solid #dddddd'>records_processed</td><td style='border: 1px solid #dddddd'>7e38094c-c3bf-418a-beb7-7033e7e8568a</td><td style='border: 1px solid #dddddd'>408.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['algorithm_used:total_variation', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>ddf7db84-b9a6-4436-93c4-e386a5adb8d7</td><td style='border: 1px solid #dddddd'>ed70ede4-5f54-491c-86c1-9845d7530447</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>e4098249-ffec-436c-b76d-0b2ffeec5bf4</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-06 13:03:38.611350+00:00</td><td style='border: 1px solid #dddddd'>confidence_drift_score</td><td style='border: 1px solid #dddddd'>7e38094c-c3bf-418a-beb7-7033e7e8568a</td><td style='border: 1px solid #dddddd'>0.1583</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.05</td><td style='border: 1px solid #dddddd'>['algorithm_used:total_variation', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>ddf7db84-b9a6-4436-93c4-e386a5adb8d7</td><td style='border: 1px solid #dddddd'>ed70ede4-5f54-491c-86c1-9845d7530447</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>e4098249-ffec-436c-b76d-0b2ffeec5bf4</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-06 13:03:38.611350+00:00</td><td style='border: 1px solid #dddddd'>records_processed</td><td style='border: 1px solid #dddddd'>7e38094c-c3bf-418a-beb7-7033e7e8568a</td><td style='border: 1px solid #dddddd'>408.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['algorithm_used:overlap_coefficient', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>ddf7db84-b9a6-4436-93c4-e386a5adb8d7</td><td style='border: 1px solid #dddddd'>ed70ede4-5f54-491c-86c1-9845d7530447</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>e4098249-ffec-436c-b76d-0b2ffeec5bf4</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-06 13:03:38.611350+00:00</td><td style='border: 1px solid #dddddd'>confidence_drift_score</td><td style='border: 1px solid #dddddd'>7e38094c-c3bf-418a-beb7-7033e7e8568a</td><td style='border: 1px solid #dddddd'>0.1598</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.05</td><td style='border: 1px solid #dddddd'>['algorithm_used:overlap_coefficient', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>ddf7db84-b9a6-4436-93c4-e386a5adb8d7</td><td style='border: 1px solid #dddddd'>ed70ede4-5f54-491c-86c1-9845d7530447</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>e4098249-ffec-436c-b76d-0b2ffeec5bf4</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-06 13:03:38.821078+00:00</td><td style='border: 1px solid #dddddd'>records_processed</td><td style='border: 1px solid #dddddd'>10619c86-1908-48ea-9301-0df6fdffad98</td><td style='border: 1px solid #dddddd'>408.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['algorithm_used:jensen_shannon', 'computed_on:payload', 'field_type:class', 'field_name:predictedLabel']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>ddf7db84-b9a6-4436-93c4-e386a5adb8d7</td><td style='border: 1px solid #dddddd'>ed70ede4-5f54-491c-86c1-9845d7530447</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>e4098249-ffec-436c-b76d-0b2ffeec5bf4</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-06 13:03:38.821078+00:00</td><td style='border: 1px solid #dddddd'>prediction_drift_score</td><td style='border: 1px solid #dddddd'>10619c86-1908-48ea-9301-0df6fdffad98</td><td style='border: 1px solid #dddddd'>0.0235</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.05</td><td style='border: 1px solid #dddddd'>['algorithm_used:jensen_shannon', 'computed_on:payload', 'field_type:class', 'field_name:predictedLabel']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>ddf7db84-b9a6-4436-93c4-e386a5adb8d7</td><td style='border: 1px solid #dddddd'>ed70ede4-5f54-491c-86c1-9845d7530447</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>e4098249-ffec-436c-b76d-0b2ffeec5bf4</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: First 10 records were displayed.\n"
     ]
    }
   ],
   "source": [
    "wos_client.monitor_instances.show_metrics(monitor_instance_id=drift_v2_monitor_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3cdf7ea4-5854-4a83-8680-f1effda282d2"
   },
   "source": [
    "## Run on-demand Fairness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "id": "1bebebf0-f118-4eab-8273-0935ef82dccc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "========================================================================\n",
      "\n",
      " Waiting for end of monitoring run c41cf7b7-1295-4235-b019-fbeae8ea3de6 \n",
      "\n",
      "========================================================================\n",
      "\n",
      "\n",
      "\n",
      "running.\n",
      "finished\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "if fairness_monitor_instance_id is not None:\n",
    "    run_details = wos_client.monitor_instances.run(monitor_instance_id=fairness_monitor_instance_id, background_mode=False).result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "id": "12629b95-2a2d-4aca-9227-7b798e82fcc5"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>5bd1947b-f9df-42ba-82d2-ebafcbe5eb42 Monitor Runs Metrics from: 2024-07-31 12:09:59.949168  till: 2024-08-07 12:09:59.949177</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower_limit</th><th style='border: 1px solid #dddddd'>upper_limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>monitor_instance_id</th><th style='border: 1px solid #dddddd'>run_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>target_id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2024-08-07 06:39:53.870393+00:00</td><td style='border: 1px solid #dddddd'>fairness_value</td><td style='border: 1px solid #dddddd'>55b17d1e-ea4a-4cd8-8d34-9d1d53927775</td><td style='border: 1px solid #dddddd'>83.146</td><td style='border: 1px solid #dddddd'>95.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['feature:Sex', 'fairness_metric_type:fairness', 'feature_value:female']</td><td style='border: 1px solid #dddddd'>fairness</td><td style='border: 1px solid #dddddd'>5bd1947b-f9df-42ba-82d2-ebafcbe5eb42</td><td style='border: 1px solid #dddddd'>c41cf7b7-1295-4235-b019-fbeae8ea3de6</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:39:53.870393+00:00</td><td style='border: 1px solid #dddddd'>fairness_value</td><td style='border: 1px solid #dddddd'>55b17d1e-ea4a-4cd8-8d34-9d1d53927775</td><td style='border: 1px solid #dddddd'>107.595</td><td style='border: 1px solid #dddddd'>95.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['feature:Age', 'fairness_metric_type:fairness', 'feature_value:18-25']</td><td style='border: 1px solid #dddddd'>fairness</td><td style='border: 1px solid #dddddd'>5bd1947b-f9df-42ba-82d2-ebafcbe5eb42</td><td style='border: 1px solid #dddddd'>c41cf7b7-1295-4235-b019-fbeae8ea3de6</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wos_client.monitor_instances.show_metrics(monitor_instance_id=fairness_monitor_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "40354858-c9b1-4d10-a1f7-225a7f1c5afa"
   },
   "source": [
    "## Run on-demand MRM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "id": "d9812595-255b-4046-842e-53488dbf7b31"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "========================================================================\n",
      "\n",
      " Waiting for end of monitoring run 3ba8c974-0e5d-4383-be2f-6130195ef4c8 \n",
      "\n",
      "========================================================================\n",
      "\n",
      "\n",
      "\n",
      "finished\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "if mrm_monitor_instance_id is not None:\n",
    "    run_details = wos_client.monitor_instances.run(monitor_instance_id=mrm_monitor_instance_id, triggered_by='user', background_mode=False).result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "id": "3c3f5422-2c19-4fdc-bd80-046b5ed35833"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>23b1f186-f599-45fb-830c-2810c18bad42 Monitor Runs Metrics from: 2024-07-31 12:10:07.374331  till: 2024-08-07 12:10:07.374337</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower_limit</th><th style='border: 1px solid #dddddd'>upper_limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>monitor_instance_id</th><th style='border: 1px solid #dddddd'>run_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>target_id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2024-08-07 06:40:01.501000+00:00</td><td style='border: 1px solid #dddddd'>fairness_score</td><td style='border: 1px solid #dddddd'>1f395c0c-8fdb-4d26-bf1b-f9acd1668d71</td><td style='border: 1px solid #dddddd'>0.83146</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>[]</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>23b1f186-f599-45fb-830c-2810c18bad42</td><td style='border: 1px solid #dddddd'>3ba8c974-0e5d-4383-be2f-6130195ef4c8</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:40:01.501000+00:00</td><td style='border: 1px solid #dddddd'>drift_score</td><td style='border: 1px solid #dddddd'>1f395c0c-8fdb-4d26-bf1b-f9acd1668d71</td><td style='border: 1px solid #dddddd'>0.07</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>[]</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>23b1f186-f599-45fb-830c-2810c18bad42</td><td style='border: 1px solid #dddddd'>3ba8c974-0e5d-4383-be2f-6130195ef4c8</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:40:01.501000+00:00</td><td style='border: 1px solid #dddddd'>tests_passed</td><td style='border: 1px solid #dddddd'>1f395c0c-8fdb-4d26-bf1b-f9acd1668d71</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>[]</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>23b1f186-f599-45fb-830c-2810c18bad42</td><td style='border: 1px solid #dddddd'>3ba8c974-0e5d-4383-be2f-6130195ef4c8</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:40:01.501000+00:00</td><td style='border: 1px solid #dddddd'>quality_score</td><td style='border: 1px solid #dddddd'>1f395c0c-8fdb-4d26-bf1b-f9acd1668d71</td><td style='border: 1px solid #dddddd'>0.3020979020979022</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>[]</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>23b1f186-f599-45fb-830c-2810c18bad42</td><td style='border: 1px solid #dddddd'>3ba8c974-0e5d-4383-be2f-6130195ef4c8</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:40:01.501000+00:00</td><td style='border: 1px solid #dddddd'>tests_run</td><td style='border: 1px solid #dddddd'>1f395c0c-8fdb-4d26-bf1b-f9acd1668d71</td><td style='border: 1px solid #dddddd'>3.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>[]</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>23b1f186-f599-45fb-830c-2810c18bad42</td><td style='border: 1px solid #dddddd'>3ba8c974-0e5d-4383-be2f-6130195ef4c8</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:40:01.501000+00:00</td><td style='border: 1px solid #dddddd'>tests_skipped</td><td style='border: 1px solid #dddddd'>1f395c0c-8fdb-4d26-bf1b-f9acd1668d71</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>[]</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>23b1f186-f599-45fb-830c-2810c18bad42</td><td style='border: 1px solid #dddddd'>3ba8c974-0e5d-4383-be2f-6130195ef4c8</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:40:01.501000+00:00</td><td style='border: 1px solid #dddddd'>tests_failed</td><td style='border: 1px solid #dddddd'>1f395c0c-8fdb-4d26-bf1b-f9acd1668d71</td><td style='border: 1px solid #dddddd'>2.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>[]</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>23b1f186-f599-45fb-830c-2810c18bad42</td><td style='border: 1px solid #dddddd'>3ba8c974-0e5d-4383-be2f-6130195ef4c8</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:37:26.104000+00:00</td><td style='border: 1px solid #dddddd'>drift_score</td><td style='border: 1px solid #dddddd'>cd6b8a45-fd97-49ca-9317-123d78db78f5</td><td style='border: 1px solid #dddddd'>0.07</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>[]</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>23b1f186-f599-45fb-830c-2810c18bad42</td><td style='border: 1px solid #dddddd'>e4a4315a-fcb7-4f16-9634-385d37ac2344</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:37:26.104000+00:00</td><td style='border: 1px solid #dddddd'>tests_passed</td><td style='border: 1px solid #dddddd'>cd6b8a45-fd97-49ca-9317-123d78db78f5</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>[]</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>23b1f186-f599-45fb-830c-2810c18bad42</td><td style='border: 1px solid #dddddd'>e4a4315a-fcb7-4f16-9634-385d37ac2344</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr><tr><td style='border: 1px solid #dddddd'>2024-08-07 06:37:26.104000+00:00</td><td style='border: 1px solid #dddddd'>quality_score</td><td style='border: 1px solid #dddddd'>cd6b8a45-fd97-49ca-9317-123d78db78f5</td><td style='border: 1px solid #dddddd'>0.3020979020979022</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>[]</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>23b1f186-f599-45fb-830c-2810c18bad42</td><td style='border: 1px solid #dddddd'>e4a4315a-fcb7-4f16-9634-385d37ac2344</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>61003e50-20d6-4ce8-a379-013a03c44229</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: First 10 records were displayed.\n"
     ]
    }
   ],
   "source": [
    "wos_client.monitor_instances.show_metrics(monitor_instance_id=mrm_monitor_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "69ff86e6-b02a-4963-941f-cfd41cd179db"
   },
   "source": [
    "## Refresh the model summary of the production subscription in the OpenScale dashboard\n",
    "\n",
    "This brings us to the end of this demo exercise. Thank you for trying it out."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}