IBM Cloud/WML/notebooks/binary/spark/Watson OpenScale and Watson ML Engine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/pmservice/ai-openscale-tutorials/raw/master/notebooks/images/banner.png\" align=\"left\" alt=\"banner\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with Watson Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook should be run in a Watson Studio project, using **Default Spark 3.3.x & Python 3.10 or Python 3.9** runtime environment. **If you are viewing this in Watson Studio and do not see the required runtime env in the upper right corner of your screen, please update the runtime now.** It requires service credentials for the following Cloud services:\n",
    "  * Watson OpenScale\n",
    "  * Watson Machine Learning V2 plans\n",
    "  * Cloud Object Storage\n",
    "  \n",
    "If you have a paid Cloud account, you may also provision a **Databases for PostgreSQL** or **Db2 Warehouse** service to take full advantage of integration with Watson Studio. If you choose not to provision this paid service, you can use the free internal PostgreSQL storage with OpenScale, but will not be able to configure continuous learning for your model.\n",
    "\n",
    "The notebook will train, create and deploy a German Credit Risk model, configure OpenScale to monitor that deployment, and inject seven days' worth of historical records and measurements for viewing in the OpenScale Insights dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Contents\n",
    "\n",
    "- [Setup](#setup)\n",
    "- [Model building and deployment](#model)\n",
    "- [OpenScale configuration](#openscale)\n",
    "- [Quality monitor and feedback logging](#quality)\n",
    "- [Fairness, drift monitoring and explanations](#fairness)\n",
    "- [Custom monitors and metrics](#custom)\n",
    "- [Historical data](#historical)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note: Samples using most latest openscale V2 client from public pypi. It does not cover followings aspects for now:\n",
    "\n",
    "- Historical payload logging \n",
    "- Historical manual labeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup <a name=\"setup\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spark check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# IF you are not using  IBM Studio to run your notebook then install following packages by uncommenting it\n",
    "#!pip install --upgrade pyspark==3.3.0 --no-cache | tail -n 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ibm-watson-openscale         3.0.39\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.1.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython3.10 -m pip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    from pyspark.sql import SparkSession\n",
    "except:\n",
    "    print('Error: Spark runtime is missing. If you are using Watson Studio change the notebook runtime to Spark.')\n",
    "    raise \n",
    "    \n",
    "!pip list | grep ibm-watson-openscale     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Package installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# IF you are not using  IBM Studio to run your notebook then install following packages by uncommenting it\n",
    "\n",
    "#!pip install pandas --no-cache | tail -n 1\n",
    "#!pip install requests --no-cache | tail -n 1\n",
    "#!pip install numpy--user --no-cache | tail -n 1\n",
    "#!pip install SciPy --no-cache | tail -n 1\n",
    "\n",
    "!pip install --upgrade ibm-watson-machine-learning --user | tail -n 1\n",
    "!pip install --upgrade \"ibm-watson-openscale~=3.0.34\" --no-cache --user | tail -n 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Provision services and configure credentials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have not already, provision an instance of IBM Watson OpenScale using the [OpenScale link in the Cloud catalog](https://cloud.ibm.com/catalog/services/watson-openscale)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your Cloud API key can be generated by going to the [**Users** section of the Cloud console](https://cloud.ibm.com/iam#/users). From that page, click your name, scroll down to the **API Keys** section, and click **Create an IBM Cloud API key**. Give your key a name and click **Create**, then copy the created key and paste it below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** You can also get OpenScale `API_KEY` using IBM CLOUD CLI.\n",
    "\n",
    "How to install IBM Cloud (bluemix) console: [instruction](https://console.bluemix.net/docs/cli/reference/ibmcloud/download_cli.html#install_use)\n",
    "\n",
    "How to get api key using console:\n",
    "```\n",
    "bx login --sso\n",
    "bx iam api-key-create 'my_key'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "CLOUD_API_KEY = \"\"\n",
    "IAM_URL=\"https://iam.ng.bluemix.net/oidc/token\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have not already, provision an instance of IBM Watson OpenScale using the [OpenScale link in the Cloud catalog](https://cloud.ibm.com/catalog/services/watson-openscale).\n",
    "\n",
    "Your Cloud API key can be generated by going to the [**Users** section of the Cloud console](https://cloud.ibm.com/iam#/users). From that page, click your name, scroll down to the **API Keys** section, and click **Create an IBM Cloud API key**. Give your key a name and click **Create**, then copy the created key, generate an IAM token using that key and paste it below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### WML credentials example with API key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "WML_CREDENTIALS = {\n",
    "                   \"url\": \"https://us-south.ml.cloud.ibm.com\",\n",
    "                   \"apikey\": CLOUD_API_KEY\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  WML credentials example using IAM_token \n",
    "\n",
    "**NOTE**: If IAM_TOKEN is used for authentication and you receive unauthorized/expired token error at any steps, please create a new token and reinitiate clients authentication."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cloud object storage details\n",
    "\n",
    "In next cells, you will need to paste some credentials to Cloud Object Storage. If you haven't worked with COS yet please visit [getting started with COS tutorial](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started). \n",
    "You can find `COS_API_KEY_ID` and `COS_RESOURCE_CRN` variables in **_Service Credentials_** in menu of your COS instance. Used COS Service Credentials must be created with _Role_ parameter set as Writer. Later training data file will be loaded to the bucket of your instance and used as training refecence in subsription.  \n",
    "`COS_ENDPOINT` variable can be found in **_Endpoint_** field of the menu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "COS_API_KEY_ID = \"***\"\n",
    "COS_RESOURCE_CRN = \"***\" # eg \"crn:v1:bluemix:public:cloud-object-storage:global:a/3bf0d9003abfb5d29761c3e97696b71c:d6f04d83-6c4f-4a62-a165-696756d63903::\"\n",
    "COS_ENDPOINT = \"***\" # Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BUCKET_NAME = \"\" #example: \"credit-risk-training-data\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial can use Databases for PostgreSQL, Db2 Warehouse, or a free internal verison of PostgreSQL to create a datamart for OpenScale.\n",
    "\n",
    "If you have previously configured OpenScale, it will use your existing datamart, and not interfere with any models you are currently monitoring. Do not update the cell below.\n",
    "\n",
    "If you do not have a paid Cloud account or would prefer not to provision this paid service, you may use the free internal PostgreSQL service with OpenScale. Do not update the cell below.\n",
    "\n",
    "To provision a new instance of Db2 Warehouse, locate [Db2 Warehouse in the Cloud catalog](https://cloud.ibm.com/catalog/services/db2-warehouse), give your service a name, and click **Create**. Once your instance is created, click the **Service Credentials** link on the left side of the screen. Click the **New credential** button, give your credentials a name, and click **Add**. Your new credentials can be accessed by clicking the **View credentials** button. Copy and paste your Db2 Warehouse credentials into the cell below.\n",
    "\n",
    "To provision a new instance of Databases for PostgreSQL, locate [Databases for PostgreSQL in the Cloud catalog](https://cloud.ibm.com/catalog/services/databases-for-postgresql), give your service a name, and click **Create**. Once your instance is created, click the **Service Credentials** link on the left side of the screen. Click the **New credential** button, give your credentials a name, and click **Add**. Your new credentials can be accessed by clicking the **View credentials** button. Copy and paste your Databases for PostgreSQL credentials into the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "DB_CREDENTIALS = None\n",
    "#DB_CREDENTIALS= {\"hostname\":\"\",\"username\":\"\",\"password\":\"\",\"database\":\"\",\"port\":\"\",\"ssl\":True,\"sslmode\":\"\",\"certificate_base64\":\"\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__If you previously configured OpenScale to use the free internal version of PostgreSQL, you can switch to a new datamart using a paid database service.__ If you would like to delete the internal PostgreSQL configuration and create a new one using service credentials supplied in the cell above, set the __KEEP_MY_INTERNAL_POSTGRES__ variable below to __False__ below. In this case, the notebook will remove your existing internal PostgreSQL datamart and create a new one with the supplied credentials. __*NO DATA MIGRATION WILL OCCUR.*__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "KEEP_MY_INTERNAL_POSTGRES = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the notebook\n",
    "\n",
    "At this point, the notebook is ready to run. You can either run the cells one at a time, or click the **Kernel** option above and select **Restart and Run All** to run all the cells."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model building and deployment <a name=\"model\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section you will learn how to train Spark MLLib model and next deploy it as web-service using Watson Machine Learning service."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the training data from github"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--  1 nelwin  staff   673K Jul 16 11:38 german_credit_data_biased_training.csv\n"
     ]
    }
   ],
   "source": [
    "from IPython.utils import io\n",
    "\n",
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/german_credit_data_biased_training.csv -O german_credit_data_biased_training.csv\n",
    "!ls -lh german_credit_data_biased_training.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/07/16 11:38:14 WARN Utils: Your hostname, Nelwins-MacBook-Pro.local resolves to a loopback address: 127.0.0.1; using 192.168.29.202 instead (on interface en0)\n",
      "24/07/16 11:38:14 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n",
      "24/07/16 11:38:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Row(CheckingStatus='0_to_200', LoanDuration=31, CreditHistory='credits_paid_to_date', LoanPurpose='other', LoanAmount=1889, ExistingSavings='100_to_500', EmploymentDuration='less_1', InstallmentPercent=3, Sex='female', OthersOnLoan='none', CurrentResidenceDuration=3, OwnsProperty='savings_insurance', Age=32, InstallmentPlans='none', Housing='own', ExistingCreditsCount=1, Job='skilled', Dependents=1, Telephone='none', ForeignWorker='yes', Risk='No Risk')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "import pandas as pd\n",
    "import json\n",
    "import datetime\n",
    "\n",
    "spark = SparkSession.builder.getOrCreate()\n",
    "pd_data = pd.read_csv(\"german_credit_data_biased_training.csv\", sep=\",\", header=0)\n",
    "df_data = spark.read.csv(path=\"german_credit_data_biased_training.csv\", sep=\",\", header=True, inferSchema=True)\n",
    "training_data_file_name = \"german_credit_data_biased_training.csv\"\n",
    "df_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- CheckingStatus: string (nullable = true)\n",
      " |-- LoanDuration: integer (nullable = true)\n",
      " |-- CreditHistory: string (nullable = true)\n",
      " |-- LoanPurpose: string (nullable = true)\n",
      " |-- LoanAmount: integer (nullable = true)\n",
      " |-- ExistingSavings: string (nullable = true)\n",
      " |-- EmploymentDuration: string (nullable = true)\n",
      " |-- InstallmentPercent: integer (nullable = true)\n",
      " |-- Sex: string (nullable = true)\n",
      " |-- OthersOnLoan: string (nullable = true)\n",
      " |-- CurrentResidenceDuration: integer (nullable = true)\n",
      " |-- OwnsProperty: string (nullable = true)\n",
      " |-- Age: integer (nullable = true)\n",
      " |-- InstallmentPlans: string (nullable = true)\n",
      " |-- Housing: string (nullable = true)\n",
      " |-- ExistingCreditsCount: integer (nullable = true)\n",
      " |-- Job: string (nullable = true)\n",
      " |-- Dependents: integer (nullable = true)\n",
      " |-- Telephone: string (nullable = true)\n",
      " |-- ForeignWorker: string (nullable = true)\n",
      " |-- Risk: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df_data.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of records: 5000\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of records: \" + str(df_data.count()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save training data to Cloud Object Storage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ibm_boto3\n",
    "from ibm_botocore.client import Config, ClientError\n",
    "\n",
    "cos_client = ibm_boto3.resource(\"s3\",\n",
    "    ibm_api_key_id=COS_API_KEY_ID,\n",
    "    ibm_service_instance_id=COS_RESOURCE_CRN,\n",
    "    ibm_auth_endpoint=\"https://iam.bluemix.net/oidc/token\",\n",
    "    config=Config(signature_version=\"oauth\"),\n",
    "    endpoint_url=COS_ENDPOINT\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(training_data_file_name, \"rb\") as file_data:\n",
    "    cos_client.Object(BUCKET_NAME, training_data_file_name).upload_fileobj(\n",
    "        Fileobj=file_data\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of records for training: 4005\n",
      "Number of records for evaluation: 995\n"
     ]
    }
   ],
   "source": [
    "spark_df = df_data\n",
    "(train_data, test_data) = spark_df.randomSplit([0.8, 0.2], 24)\n",
    "\n",
    "MODEL_NAME = \"Spark German Risk Model - Final\"\n",
    "DEPLOYMENT_NAME = \"Spark German Risk Deployment - Final\"\n",
    "\n",
    "print(\"Number of records for training: \" + str(train_data.count()))\n",
    "print(\"Number of records for evaluation: \" + str(test_data.count()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below creates a Random Forest Classifier with Spark, setting up string indexers for the categorical features and the label column. Finally, this notebook creates a pipeline including the indexers and the model, and does an initial Area Under ROC evaluation of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import OneHotEncoder, StringIndexer, IndexToString, VectorAssembler\n",
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "from pyspark.ml import Pipeline, Model\n",
    "from pyspark.ml.feature import SQLTransformer\n",
    "\n",
    "features = [x for x in spark_df.columns if x != 'Risk']\n",
    "categorical_features = ['CheckingStatus', 'CreditHistory', 'LoanPurpose', 'ExistingSavings', 'EmploymentDuration', 'Sex', 'OthersOnLoan', 'OwnsProperty', 'InstallmentPlans', 'Housing', 'Job', 'Telephone', 'ForeignWorker']\n",
    "categorical_num_features = [x + '_IX' for x in categorical_features]\n",
    "si_list = [StringIndexer(inputCol=x, outputCol=y) for x, y in zip(categorical_features, categorical_num_features)]\n",
    "va_features = VectorAssembler(inputCols=categorical_num_features + [x for x in features if x not in categorical_features], outputCol=\"features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "si_label = StringIndexer(inputCol=\"Risk\", outputCol=\"label\").fit(spark_df)\n",
    "label_converter = IndexToString(inputCol=\"prediction\", outputCol=\"predictedLabel\", labels=si_label.labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.classification import RandomForestClassifier\n",
    "\n",
    "classifier = RandomForestClassifier(featuresCol=\"features\")\n",
    "feature_filter = SQLTransformer(statement=\"SELECT * FROM __THIS__\")\n",
    "pipeline = Pipeline(stages= si_list + [si_label, va_features, classifier, label_converter, feature_filter])\n",
    "model_rf = pipeline.fit(train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** If you want filter features from model output please replace **`*`** with feature names to be retained in **`SQLTransformer`** statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/07/16 11:38:31 WARN SparkStringUtils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "areaUnderROC = 0.722391\n"
     ]
    }
   ],
   "source": [
    "predictions = model_rf.transform(test_data)\n",
    "evaluatorDT = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\")\n",
    "area_under_curve = evaluatorDT.evaluate(predictions)\n",
    "\n",
    "print(\"areaUnderROC = %g\" % area_under_curve)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Publish the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, the notebook uses the supplied Watson Machine Learning credentials to save the model (including the pipeline) to the WML instance. Previous versions of the model are removed so that the notebook can be run again, resetting all data for another demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.0.360'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "from ibm_watson_machine_learning import APIClient\n",
    "\n",
    "wml_client = APIClient(WML_CREDENTIALS)\n",
    "wml_client.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing all the available spaces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------  -------------------------------------------------------------------  ------------------------\n",
      "ID                                    NAME                                                                 CREATED\n",
      "be45ab4c-1fb7-440c-9b03-2909067e45e0  Automotive Demo - Quality report summarization                       2024-05-27T13:13:27.233Z\n",
      "e0ee6250-7ef6-42c3-8ffa-350d9b0df578  pre-prod-space                                                       2024-02-28T07:35:30.368Z\n",
      "63c5982f-7160-41c4-86f7-1310a8ab32cb  prompt-space                                                         2024-01-15T19:29:21.535Z\n",
      "f04e0e73-a1b7-4ae9-a08d-7e16add4fe08  llm. space                                                           2023-11-24T13:57:05.167Z\n",
      "d1afbea3-e899-4ed3-b9a6-0686751508c3  wml                                                                  2023-09-21T07:06:42.577Z\n",
      "6f7c3969-6d3f-4f9a-b97a-b534f4e4fef3  AutoAIDemo                                                           2023-08-25T02:50:27.113Z\n",
      "0b7992c2-3991-4145-a5ba-d5b428261171  openscale-express-path-preprod-80e6093f-5acf-4eb7-9da6-7ba9bf56a929  2023-08-16T06:59:18.538Z\n",
      "3226c381-5ae0-4bc4-b306-fc638c785e47  openscale-express-path-80e6093f-5acf-4eb7-9da6-7ba9bf56a929          2023-08-16T06:58:57.332Z\n",
      "------------------------------------  -------------------------------------------------------------------  ------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>NAME</th>\n",
       "      <th>CREATED</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>be45ab4c-1fb7-440c-9b03-2909067e45e0</td>\n",
       "      <td>Automotive Demo - Quality report summarization</td>\n",
       "      <td>2024-05-27T13:13:27.233Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>e0ee6250-7ef6-42c3-8ffa-350d9b0df578</td>\n",
       "      <td>pre-prod-space</td>\n",
       "      <td>2024-02-28T07:35:30.368Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>63c5982f-7160-41c4-86f7-1310a8ab32cb</td>\n",
       "      <td>prompt-space</td>\n",
       "      <td>2024-01-15T19:29:21.535Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>f04e0e73-a1b7-4ae9-a08d-7e16add4fe08</td>\n",
       "      <td>llm. space</td>\n",
       "      <td>2023-11-24T13:57:05.167Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>d1afbea3-e899-4ed3-b9a6-0686751508c3</td>\n",
       "      <td>wml</td>\n",
       "      <td>2023-09-21T07:06:42.577Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6f7c3969-6d3f-4f9a-b97a-b534f4e4fef3</td>\n",
       "      <td>AutoAIDemo</td>\n",
       "      <td>2023-08-25T02:50:27.113Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0b7992c2-3991-4145-a5ba-d5b428261171</td>\n",
       "      <td>openscale-express-path-preprod-80e6093f-5acf-4...</td>\n",
       "      <td>2023-08-16T06:59:18.538Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3226c381-5ae0-4bc4-b306-fc638c785e47</td>\n",
       "      <td>openscale-express-path-80e6093f-5acf-4eb7-9da6...</td>\n",
       "      <td>2023-08-16T06:58:57.332Z</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  \\\n",
       "0  be45ab4c-1fb7-440c-9b03-2909067e45e0   \n",
       "1  e0ee6250-7ef6-42c3-8ffa-350d9b0df578   \n",
       "2  63c5982f-7160-41c4-86f7-1310a8ab32cb   \n",
       "3  f04e0e73-a1b7-4ae9-a08d-7e16add4fe08   \n",
       "4  d1afbea3-e899-4ed3-b9a6-0686751508c3   \n",
       "5  6f7c3969-6d3f-4f9a-b97a-b534f4e4fef3   \n",
       "6  0b7992c2-3991-4145-a5ba-d5b428261171   \n",
       "7  3226c381-5ae0-4bc4-b306-fc638c785e47   \n",
       "\n",
       "                                                NAME                   CREATED  \n",
       "0     Automotive Demo - Quality report summarization  2024-05-27T13:13:27.233Z  \n",
       "1                                     pre-prod-space  2024-02-28T07:35:30.368Z  \n",
       "2                                       prompt-space  2024-01-15T19:29:21.535Z  \n",
       "3                                         llm. space  2023-11-24T13:57:05.167Z  \n",
       "4                                                wml  2023-09-21T07:06:42.577Z  \n",
       "5                                         AutoAIDemo  2023-08-25T02:50:27.113Z  \n",
       "6  openscale-express-path-preprod-80e6093f-5acf-4...  2023-08-16T06:59:18.538Z  \n",
       "7  openscale-express-path-80e6093f-5acf-4eb7-9da6...  2023-08-16T06:58:57.332Z  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wml_client.spaces.list(limit=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'SUCCESS'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "WML_SPACE_ID='' # use space id here\n",
    "wml_client.set.default_space(WML_SPACE_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove existing model and deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------  ----------------------------------------  ------------------------  ---------  ----------  ----------------\n",
      "ID                                    NAME                                      CREATED                   TYPE       SPEC_STATE  SPEC_REPLACEMENT\n",
      "3c89d675-9c16-4f02-8eb1-a7744b6b1e4b  Spark German Risk Model - Training stats  2024-07-15T10:04:40.002Z  mllib_3.3  supported\n",
      "------------------------------------  ----------------------------------------  ------------------------  ---------  ----------  ----------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>NAME</th>\n",
       "      <th>CREATED</th>\n",
       "      <th>TYPE</th>\n",
       "      <th>SPEC_STATE</th>\n",
       "      <th>SPEC_REPLACEMENT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3c89d675-9c16-4f02-8eb1-a7744b6b1e4b</td>\n",
       "      <td>Spark German Risk Model - Training stats</td>\n",
       "      <td>2024-07-15T10:04:40.002Z</td>\n",
       "      <td>mllib_3.3</td>\n",
       "      <td>supported</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  \\\n",
       "0  3c89d675-9c16-4f02-8eb1-a7744b6b1e4b   \n",
       "\n",
       "                                       NAME                   CREATED  \\\n",
       "0  Spark German Risk Model - Training stats  2024-07-15T10:04:40.002Z   \n",
       "\n",
       "        TYPE SPEC_STATE SPEC_REPLACEMENT  \n",
       "0  mllib_3.3  supported                   "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deployments_list = wml_client.deployments.get_details()\n",
    "for deployment in deployments_list[\"resources\"]:\n",
    "    model_id = deployment[\"entity\"][\"asset\"][\"id\"]\n",
    "    deployment_id = deployment[\"metadata\"][\"id\"]\n",
    "    if deployment[\"metadata\"][\"name\"] == DEPLOYMENT_NAME:\n",
    "        print(\"Deleting deployment id\", deployment_id)\n",
    "        wml_client.deployments.delete(deployment_id)\n",
    "        print(\"Deleting model id\", model_id)\n",
    "        wml_client.repository.delete(model_id)\n",
    "wml_client.repository.list_models()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating connections...\n",
      "SUCCESS\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2f11d26a-3a22-4f25-9b13-62ff00d4049a'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Create connections to a COS bucket\n",
    "datasource_type = wml_client.connections.get_datasource_type_uid_by_name('bluemixcloudobjectstorage')\n",
    "conn_meta_props= {\n",
    "    wml_client.connections.ConfigurationMetaNames.NAME: \"Connection My COS \",\n",
    "    wml_client.connections.ConfigurationMetaNames.DATASOURCE_TYPE: datasource_type,\n",
    "    wml_client.connections.ConfigurationMetaNames.DESCRIPTION: \"Connection to my COS\",\n",
    "    wml_client.connections.ConfigurationMetaNames.PROPERTIES: {\n",
    "        'bucket': BUCKET_NAME,\n",
    "        'api_key': CLOUD_API_KEY,\n",
    "        'resource_instance_id': COS_RESOURCE_CRN,\n",
    "        'iam_url': IAM_URL,\n",
    "        'url': COS_ENDPOINT\n",
    "    }\n",
    "}\n",
    "\n",
    "conn_details = wml_client.connections.create(meta_props=conn_meta_props)\n",
    "connection_id = wml_client.connections.get_uid(conn_details)\n",
    "connection_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_data_references = [\n",
    "                {\n",
    "                    \"id\": \"German Credit Risk\", \n",
    "                    \"type\": \"connection_asset\",\n",
    "                    \"connection\": {\n",
    "                        \"id\": connection_id,\n",
    "                        \"href\": \"/v2/connections/\" + connection_id + \"?space_id=\" + WML_SPACE_ID\n",
    "                        \n",
    "                    },\n",
    "                    \"location\": {\n",
    "                        \"bucket\": BUCKET_NAME,\n",
    "                        \"file_name\": training_data_file_name\n",
    "                    },\n",
    "                    \"schema\": {\n",
    "                    \"id\": \"training_schema\",\n",
    "                    \"fields\": [\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"CheckingStatus\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"string\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"LoanDuration\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"integer\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"CreditHistory\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"string\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"LoanPurpose\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"string\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"LoanAmount\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"integer\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"ExistingSavings\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"string\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"EmploymentDuration\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"string\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"InstallmentPercent\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"integer\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"Sex\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"string\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"OthersOnLoan\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"string\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"CurrentResidenceDuration\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"integer\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"OwnsProperty\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"string\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"Age\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"integer\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"InstallmentPlans\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"string\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"Housing\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"string\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"ExistingCreditsCount\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"integer\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"Job\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"string\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"Dependents\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"integer\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"Telephone\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"string\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {},\n",
    "                        \"name\": \"ForeignWorker\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"string\"\n",
    "                      },\n",
    "                      {\n",
    "                        \"metadata\": {\n",
    "                          \"modeling_role\": \"target\"\n",
    "                        },\n",
    "                        \"name\": \"Risk\",\n",
    "                        \"nullable\": True,\n",
    "                        \"type\": \"string\"\n",
    "                      }\n",
    "                    ]\n",
    "                  }\n",
    "                }\n",
    "            ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Software Specification ID: d11f2434-4fc7-58b7-8a62-755da64fdaf8\n"
     ]
    }
   ],
   "source": [
    "software_spec_uid = wml_client.software_specifications.get_id_by_name(\"spark-mllib_3.3\")\n",
    "print(\"Software Specification ID: {}\".format(software_spec_uid))\n",
    "model_props = {\n",
    "        wml_client._models.ConfigurationMetaNames.NAME:\"{}\".format(MODEL_NAME),\n",
    "        wml_client._models.ConfigurationMetaNames.TYPE: \"mllib_3.3\",\n",
    "        wml_client._models.ConfigurationMetaNames.SOFTWARE_SPEC_UID: software_spec_uid,\n",
    "        wml_client._models.ConfigurationMetaNames.TRAINING_DATA_REFERENCES: training_data_references,\n",
    "        wml_client._models.ConfigurationMetaNames.LABEL_FIELD: \"Risk\",\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Storing model ...\n",
      "Done\n",
      "Model ID: ce0b71a4-ecee-4e0d-b51a-123e0fad4acd\n"
     ]
    }
   ],
   "source": [
    "print(\"Storing model ...\")\n",
    "published_model_details = wml_client.repository.store_model(\n",
    "    model=model_rf, \n",
    "    meta_props=model_props, \n",
    "    training_data=train_data, \n",
    "    pipeline=pipeline)\n",
    "\n",
    "model_uid = wml_client.repository.get_model_id(published_model_details)\n",
    "print(\"Done\")\n",
    "print(\"Model ID: {}\".format(model_uid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next section of the notebook deploys the model as a RESTful web service in Watson Machine Learning. The deployed model will have a scoring URL you can use to send data to the model for predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "Synchronous deployment creation for uid: 'ce0b71a4-ecee-4e0d-b51a-123e0fad4acd' started\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "\n",
      "initializing\n",
      "Note: online_url and serving_urls are deprecated and will be removed in a future release. Use inference instead.\n",
      ".\n",
      "ready\n",
      "\n",
      "\n",
      "------------------------------------------------------------------------------------------------\n",
      "Successfully finished deployment creation, deployment_uid='db664cd4-2467-4975-b314-e24115c37ea5'\n",
      "------------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "Scoring URL:https://us-south.ml.cloud.ibm.com/ml/v4/deployments/db664cd4-2467-4975-b314-e24115c37ea5/predictions\n",
      "Model id: ce0b71a4-ecee-4e0d-b51a-123e0fad4acd\n",
      "Deployment id: db664cd4-2467-4975-b314-e24115c37ea5\n"
     ]
    }
   ],
   "source": [
    "deployment_details = wml_client.deployments.create(\n",
    "    model_uid, \n",
    "    meta_props={\n",
    "        wml_client.deployments.ConfigurationMetaNames.NAME: \"{}\".format(DEPLOYMENT_NAME),\n",
    "        wml_client.deployments.ConfigurationMetaNames.ONLINE: {}\n",
    "    }\n",
    ")\n",
    "scoring_url = wml_client.deployments.get_scoring_href(deployment_details)\n",
    "deployment_uid=wml_client.deployments.get_uid(deployment_details)\n",
    "\n",
    "print(\"Scoring URL:\" + scoring_url)\n",
    "print(\"Model id: {}\".format(model_uid))\n",
    "print(\"Deployment id: {}\".format(deployment_uid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample scoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fields = [\"CheckingStatus\", \"LoanDuration\", \"CreditHistory\", \"LoanPurpose\", \"LoanAmount\", \"ExistingSavings\",\n",
    "                  \"EmploymentDuration\", \"InstallmentPercent\", \"Sex\", \"OthersOnLoan\", \"CurrentResidenceDuration\",\n",
    "                  \"OwnsProperty\", \"Age\", \"InstallmentPlans\", \"Housing\", \"ExistingCreditsCount\", \"Job\", \"Dependents\",\n",
    "                  \"Telephone\", \"ForeignWorker\"]\n",
    "values = [\n",
    "            [\"no_checking\", 13, \"credits_paid_to_date\", \"car_new\", 1343, \"100_to_500\", \"1_to_4\", 2, \"female\", \"none\", 3,\n",
    "             \"savings_insurance\", 46, \"none\", \"own\", 2, \"skilled\", 1, \"none\", \"yes\"],\n",
    "            [\"no_checking\", 24, \"prior_payments_delayed\", \"furniture\", 4567, \"500_to_1000\", \"1_to_4\", 4, \"male\", \"none\",\n",
    "             4, \"savings_insurance\", 36, \"none\", \"free\", 2, \"management_self-employed\", 1, \"none\", \"yes\"],\n",
    "        ]\n",
    "\n",
    "scoring_payload = {\"input_data\": [{\"fields\": fields, \"values\": values}]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'predictions': [{'fields': ['CheckingStatus',\n",
       "    'LoanDuration',\n",
       "    'CreditHistory',\n",
       "    'LoanPurpose',\n",
       "    'LoanAmount',\n",
       "    'ExistingSavings',\n",
       "    'EmploymentDuration',\n",
       "    'InstallmentPercent',\n",
       "    'Sex',\n",
       "    'OthersOnLoan',\n",
       "    'CurrentResidenceDuration',\n",
       "    'OwnsProperty',\n",
       "    'Age',\n",
       "    'InstallmentPlans',\n",
       "    'Housing',\n",
       "    'ExistingCreditsCount',\n",
       "    'Job',\n",
       "    'Dependents',\n",
       "    'Telephone',\n",
       "    'ForeignWorker',\n",
       "    'CheckingStatus_IX',\n",
       "    'CreditHistory_IX',\n",
       "    'LoanPurpose_IX',\n",
       "    'ExistingSavings_IX',\n",
       "    'EmploymentDuration_IX',\n",
       "    'Sex_IX',\n",
       "    'OthersOnLoan_IX',\n",
       "    'OwnsProperty_IX',\n",
       "    'InstallmentPlans_IX',\n",
       "    'Housing_IX',\n",
       "    'Job_IX',\n",
       "    'Telephone_IX',\n",
       "    'ForeignWorker_IX',\n",
       "    'features',\n",
       "    'rawPrediction',\n",
       "    'probability',\n",
       "    'prediction',\n",
       "    'predictedLabel'],\n",
       "   'values': [['no_checking',\n",
       "     13,\n",
       "     'credits_paid_to_date',\n",
       "     'car_new',\n",
       "     1343,\n",
       "     '100_to_500',\n",
       "     '1_to_4',\n",
       "     2,\n",
       "     'female',\n",
       "     'none',\n",
       "     3,\n",
       "     'savings_insurance',\n",
       "     46,\n",
       "     'none',\n",
       "     'own',\n",
       "     2,\n",
       "     'skilled',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     [20,\n",
       "      [1, 3, 5, 13, 14, 15, 16, 17, 18, 19],\n",
       "      [1.0, 1.0, 1.0, 13.0, 1343.0, 2.0, 3.0, 46.0, 2.0, 1.0]],\n",
       "     [12.979298576936431, 7.0207014230635725],\n",
       "     [0.6489649288468214, 0.35103507115317856],\n",
       "     0.0,\n",
       "     'No Risk'],\n",
       "    ['no_checking',\n",
       "     24,\n",
       "     'prior_payments_delayed',\n",
       "     'furniture',\n",
       "     4567,\n",
       "     '500_to_1000',\n",
       "     '1_to_4',\n",
       "     4,\n",
       "     'male',\n",
       "     'none',\n",
       "     4,\n",
       "     'savings_insurance',\n",
       "     36,\n",
       "     'none',\n",
       "     'free',\n",
       "     2,\n",
       "     'management_self-employed',\n",
       "     1,\n",
       "     'none',\n",
       "     'yes',\n",
       "     0.0,\n",
       "     0.0,\n",
       "     1.0,\n",
       "     2.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     2.0,\n",
       "     2.0,\n",
       "     0.0,\n",
       "     0.0,\n",
       "     [20,\n",
       "      [2, 3, 9, 10, 13, 14, 15, 16, 17, 18, 19],\n",
       "      [1.0, 2.0, 2.0, 2.0, 24.0, 4567.0, 4.0, 4.0, 36.0, 2.0, 1.0]],\n",
       "     [13.137817951142797, 6.862182048857208],\n",
       "     [0.6568908975571397, 0.34310910244286036],\n",
       "     0.0,\n",
       "     'No Risk']]}]}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scoring_response = wml_client.deployments.score(deployment_uid, scoring_payload)\n",
    "scoring_response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configure OpenScale <a name=\"openscale\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The notebook will now import the necessary libraries and set up a Python OpenScale client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3.0.39'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from ibm_cloud_sdk_core.authenticators import IAMAuthenticator,BearerTokenAuthenticator\n",
    "\n",
    "from ibm_watson_openscale import *\n",
    "from ibm_watson_openscale.supporting_classes.enums import *\n",
    "from ibm_watson_openscale.supporting_classes import *\n",
    "\n",
    "\n",
    "authenticator = IAMAuthenticator(apikey=CLOUD_API_KEY)\n",
    "#authenticator = BearerTokenAuthenticator(bearer_token=IAM_TOKEN) ## uncomment this line if using IAM token to authenticate\n",
    "wos_client = APIClient(authenticator=authenticator)\n",
    "wos_client.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create schema and datamart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up datamart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Watson OpenScale uses a database to store payload logs and calculated metrics. If database credentials were **not** supplied above, the notebook will use the free, internal lite database. If database credentials were supplied, the datamart will be created there **unless** there is an existing datamart **and** the **KEEP_MY_INTERNAL_POSTGRES** variable is set to **True**. If an OpenScale datamart exists in Db2 or PostgreSQL, the existing datamart will be used and no data will be overwritten.\n",
    "\n",
    "Prior instances of the German Credit model will be removed from OpenScale monitoring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Data Marts</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>description</th><th style='border: 1px solid #dddddd'>internal_database</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>AIOSFASTPATH-80E6093F-5ACF-4EB7-9DA6-7BA9BF56A929</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>True</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-05-16 06:09:07.089000+00:00</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wos_client.data_marts.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using existing datamart 80e6093f-5acf-4eb7-9da6-7ba9bf56a929\n"
     ]
    }
   ],
   "source": [
    "data_marts = wos_client.data_marts.list().result.data_marts\n",
    "if len(data_marts) == 0:\n",
    "    if DB_CREDENTIALS is not None:\n",
    "        if SCHEMA_NAME is None: \n",
    "            print(\"Please specify the SCHEMA_NAME and rerun the cell\")\n",
    "\n",
    "        print('Setting up external datamart')\n",
    "        added_data_mart_result = wos_client.data_marts.add(\n",
    "                background_mode=False,\n",
    "                name=\"WOS Data Mart\",\n",
    "                description=\"Data Mart created by WOS tutorial notebook\",\n",
    "                database_configuration=DatabaseConfigurationRequest(\n",
    "                  database_type=DatabaseType.POSTGRESQL,\n",
    "                    credentials=PrimaryStorageCredentialsLong(\n",
    "                        hostname=DB_CREDENTIALS['hostname'],\n",
    "                        username=DB_CREDENTIALS['username'],\n",
    "                        password=DB_CREDENTIALS['password'],\n",
    "                        db=DB_CREDENTIALS['database'],\n",
    "                        port=DB_CREDENTIALS['port'],\n",
    "                        ssl=True,\n",
    "                        sslmode=DB_CREDENTIALS['sslmode'],\n",
    "                        certificate_base64=DB_CREDENTIALS['certificate_base64']\n",
    "                    ),\n",
    "                    location=LocationSchemaName(\n",
    "                        schema_name= SCHEMA_NAME\n",
    "                    )\n",
    "                )\n",
    "             ).result\n",
    "    else:\n",
    "        print('Setting up internal datamart')\n",
    "        added_data_mart_result = wos_client.data_marts.add(\n",
    "                background_mode=False,\n",
    "                name=\"WOS Data Mart\",\n",
    "                description=\"Data Mart created by WOS tutorial notebook\", \n",
    "                internal_database = True).result\n",
    "        \n",
    "    data_mart_id = added_data_mart_result.metadata.id\n",
    "    \n",
    "else:\n",
    "    data_mart_id=data_marts[0].metadata.id\n",
    "    print('Using existing datamart {}'.format(data_mart_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove existing service provider connected with used  WML instance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Multiple service providers for the same engine instance are avaiable in Watson OpenScale. To avoid multiple service providers of used WML instance in the tutorial notebook the following code deletes existing service provder(s) and then adds new one. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SERVICE_PROVIDER_NAME = \"Watson Machine Learning V2\"\n",
    "SERVICE_PROVIDER_DESCRIPTION = \"Added by tutorial WOS notebook.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "service_providers = wos_client.service_providers.list().result.service_providers\n",
    "for service_provider in service_providers:\n",
    "    service_instance_name = service_provider.entity.name\n",
    "    if service_instance_name == SERVICE_PROVIDER_NAME:\n",
    "        service_provider_id = service_provider.metadata.id\n",
    "        wos_client.service_providers.delete(service_provider_id)\n",
    "        print(\"Deleted existing service_provider for WML instance: {}\".format(service_provider_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add service provider"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Watson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** You can bind more than one engine instance if needed by calling `wos_client.service_providers.add` method. Next, you can refer to particular service provider using `service_provider_id`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "=================================================================================\n",
      "\n",
      " Waiting for end of adding service provider 6a38b72c-4aae-4fb0-b786-9b9edacf64bf \n",
      "\n",
      "=================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "-----------------------------------------------\n",
      " Successfully finished adding service provider \n",
      "-----------------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "added_service_provider_result = wos_client.service_providers.add(\n",
    "        name=SERVICE_PROVIDER_NAME,\n",
    "        description=SERVICE_PROVIDER_DESCRIPTION,\n",
    "        service_type=ServiceTypes.WATSON_MACHINE_LEARNING,\n",
    "        deployment_space_id = WML_SPACE_ID,\n",
    "        operational_space_id = \"production\",\n",
    "        credentials=WMLCredentialsCloud(\n",
    "            apikey=CLOUD_API_KEY,      ## use `apikey=IAM_TOKEN` if using IAM_TOKEN to initiate client\n",
    "            url=WML_CREDENTIALS[\"url\"],\n",
    "            instance_id=None\n",
    "        ),\n",
    "        background_mode=False\n",
    "    ).result\n",
    "service_provider_id = added_service_provider_result.metadata.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Service Providers</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>instance_id</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>service_type</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>89cbf463-87d9-4032-9c0c-e371dd68d156</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>Watson Machine Learning V2-2</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-07-16 06:11:13.858000+00:00</td><td style='border: 1px solid #dddddd'>6a38b72c-4aae-4fb0-b786-9b9edacf64bf</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>Azure Machine Learning</td><td style='border: 1px solid #dddddd'>azure_machine_learning</td><td style='border: 1px solid #dddddd'>2024-07-15 11:18:22.795000+00:00</td><td style='border: 1px solid #dddddd'>db2618f1-ef2b-4251-a22c-bd347702672d</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>AWS Machine Learning</td><td style='border: 1px solid #dddddd'>amazon_sagemaker</td><td style='border: 1px solid #dddddd'>2024-07-11 09:29:39.272000+00:00</td><td style='border: 1px solid #dddddd'>4fa5625b-5401-4a31-a423-d480ea184cda</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>OpenScale Headless Service Provider</td><td style='border: 1px solid #dddddd'>custom_machine_learning</td><td style='border: 1px solid #dddddd'>2024-07-05 13:55:20.532000+00:00</td><td style='border: 1px solid #dddddd'>0bf0f943-24cf-487e-b971-4a6f586effc0</td></tr><tr><td style='border: 1px solid #dddddd'>89cbf463-87d9-4032-9c0c-e371dd68d156</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>aWatson Machine Learning V2_amzon</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-06-06 06:03:03.459000+00:00</td><td style='border: 1px solid #dddddd'>84265de0-8f4b-4257-8743-7c2525e86b00</td></tr><tr><td style='border: 1px solid #dddddd'>89cbf463-87d9-4032-9c0c-e371dd68d156</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>Watson Machine Learning V2</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-06-02 08:24:57.362000+00:00</td><td style='border: 1px solid #dddddd'>e13bfbbb-e639-4875-bbd5-6d2de8e834d0</td></tr><tr><td style='border: 1px solid #dddddd'>89cbf463-87d9-4032-9c0c-e371dd68d156</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>WOS ExpressPath WML pre_production binding</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-05-16 06:09:28.464000+00:00</td><td style='border: 1px solid #dddddd'>19fc94cc-9ed2-4ac6-a98e-1acf1a9e729e</td></tr><tr><td style='border: 1px solid #dddddd'>89cbf463-87d9-4032-9c0c-e371dd68d156</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>WOS ExpressPath WML production binding</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-05-16 06:09:12.960000+00:00</td><td style='border: 1px solid #dddddd'>7ef8cc5f-65b1-4f83-9d1b-d2d508420958</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wos_client.service_providers.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'metadata': {'guid': '5b644de6-5b83-4930-ac85-c600b2c0ac4f',\n",
       "  'url': 'https://us-south.ml.cloud.ibm.com/ml/v4/deployments/5b644de6-5b83-4930-ac85-c600b2c0ac4f?space_id=63c5982f-7160-41c4-86f7-1310a8ab32cb',\n",
       "  'created_at': '2024-07-15T10:04:51.382Z',\n",
       "  'modified_at': '2024-07-15T10:04:51.382Z'},\n",
       " 'entity': {'name': 'Spark German Risk Deployment - Training stats',\n",
       "  'type': 'online',\n",
       "  'scoring_endpoint': {'url': 'https://us-south.ml.cloud.ibm.com/ml/v4/deployments/5b644de6-5b83-4930-ac85-c600b2c0ac4f/predictions'},\n",
       "  'asset': {},\n",
       "  'asset_properties': {}}}"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "asset_deployment_details = wos_client.service_providers.list_assets(data_mart_id=data_mart_id, service_provider_id=service_provider_id, deployment_space_id = WML_SPACE_ID).result['resources'][0]\n",
    "asset_deployment_details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_asset_details_from_deployment=wos_client.service_providers.get_deployment_asset(data_mart_id=data_mart_id,service_provider_id=service_provider_id,deployment_id=deployment_uid,deployment_space_id=WML_SPACE_ID)\n",
    "model_asset_details_from_deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subscriptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove existing credit risk subscriptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code removes previous subscriptions to the German Credit model to refresh the monitors with the new model and new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Subscriptions</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>asset_id</th><th style='border: 1px solid #dddddd'>asset_type</th><th style='border: 1px solid #dddddd'>asset_name</th><th style='border: 1px solid #dddddd'>data_mart_id</th><th style='border: 1px solid #dddddd'>deployment_id</th><th style='border: 1px solid #dddddd'>deployment_name</th><th style='border: 1px solid #dddddd'>service_provider_id</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>085460ef94636166aea5800e9ea26168</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>GermanCreditRisk.2019.1.9.10.41.58.611</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>GermanCreditRisk.2019.1.9.10.41.58.611</td><td style='border: 1px solid #dddddd'>db2618f1-ef2b-4251-a22c-bd347702672d</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-15 11:20:20.209000+00:00</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td></tr><tr><td style='border: 1px solid #dddddd'>98783e76b0aedefc06c70bd74fa2356e</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Credit-risk-linear-learner-2021-06-24-05-32</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>cc7acf711b19c689d67e7b2a01b88ca2</td><td style='border: 1px solid #dddddd'>Credit-risk-endpoint-scoring-2021-06-24-05-32</td><td style='border: 1px solid #dddddd'>4fa5625b-5401-4a31-a423-d480ea184cda</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-11 09:30:10.069000+00:00</td><td style='border: 1px solid #dddddd'>95e480af-2fec-43ef-b34e-b3a24572b35d</td></tr><tr><td style='border: 1px solid #dddddd'>d4259f6b-e5a1-4a33-a862-d2788e5289ef</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>[asset] GCR Headless Subscription</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>aa8bd973-d680-496b-923d-f9bcbcc0bc7a</td><td style='border: 1px solid #dddddd'>[asset] GCR Headless Subscription</td><td style='border: 1px solid #dddddd'>0bf0f943-24cf-487e-b971-4a6f586effc0</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-05 13:57:01.523000+00:00</td><td style='border: 1px solid #dddddd'>345f4646-871a-4e0c-b30f-abcf9a83ea71</td></tr><tr><td style='border: 1px solid #dddddd'>ad45394c-0cfb-403c-81f6-a20f44cded7a</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Spark German Risk Model - Final</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>8db12c5c-94cf-486e-937a-6f17dadf8a56</td><td style='border: 1px solid #dddddd'>New - Spark German Risk Deployment - Final</td><td style='border: 1px solid #dddddd'>e13bfbbb-e639-4875-bbd5-6d2de8e834d0</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-04 09:15:28.269000+00:00</td><td style='border: 1px solid #dddddd'>59622413-a37f-4e76-a4d5-46967f3cf12b</td></tr><tr><td style='border: 1px solid #dddddd'>43870a88-5fb5-425a-9099-078b0469dd2a</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Amazon Mobile Review Classifier</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>8edfc920-a4bc-47cd-83fd-9e9674476dec</td><td style='border: 1px solid #dddddd'>Amazon Mobile Review Classifier deployment2</td><td style='border: 1px solid #dddddd'>84265de0-8f4b-4257-8743-7c2525e86b00</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-06-06 06:04:06.732000+00:00</td><td style='border: 1px solid #dddddd'>7828e5f6-9cc7-4cc6-b20b-7bf844c38ccc</td></tr><tr><td style='border: 1px solid #dddddd'>699a18fc-1d52-4a1b-9a25-87d9362cd3dc</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Scikit German Risk Model WML V4</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>2de6645d-c6a9-42ae-a884-4e2bb646ad33</td><td style='border: 1px solid #dddddd'>Scikit German Risk Deployment WML V4-1</td><td style='border: 1px solid #dddddd'>e13bfbbb-e639-4875-bbd5-6d2de8e834d0</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-06-02 08:26:51.212000+00:00</td><td style='border: 1px solid #dddddd'>91a07f4c-50b6-426d-93da-94438a7edc9a</td></tr><tr><td style='border: 1px solid #dddddd'>a7fc7ed9-f000-4f27-988a-54fbc8e397e6</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>GermanCreditRiskModelChallenger</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>38059feb-e4c5-4037-90ea-a18f77a8ec5b</td><td style='border: 1px solid #dddddd'>GermanCreditRiskModelChallenger</td><td style='border: 1px solid #dddddd'>19fc94cc-9ed2-4ac6-a98e-1acf1a9e729e</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-05-16 06:09:55.832000+00:00</td><td style='border: 1px solid #dddddd'>81044f9c-27a8-42f2-90be-9cef8b33fe39</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wos_client.subscriptions.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code removes previous subscriptions to the German Credit model to refresh the monitors with the new model and new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "subscriptions = wos_client.subscriptions.list().result.subscriptions\n",
    "for subscription in subscriptions:\n",
    "    sub_model_id = subscription.entity.asset.asset_id\n",
    "    if sub_model_id == model_uid:\n",
    "        wos_client.subscriptions.delete(subscription.metadata.id)\n",
    "        print('Deleted existing subscription for model', sub_model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code creates the model subscription in OpenScale using the Python client API. Note that we need to provide the model unique identifier, and some information about the model itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This code creates the model subscription in OpenScale using the Python client API. Note that we need to provide the model unique identifier, and some information about the model itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note :\n",
    "1) User can add `training_data_schema` as an optional parameter as part of  AssetPropertiesRequest while creating subscription.\n",
    "2) If not added, schemas will get added/updated after first payload logging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# training_data_schema=SparkStruct.from_dict(model_asset_details_from_deployment[\"entity\"][\"asset_properties\"][\"training_data_schema\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watson_openscale.base_classes.watson_open_scale_v2 import ScoringEndpointRequest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "=============================================================================\n",
      "\n",
      " Waiting for end of adding subscription f1a960ef-b474-43f9-8621-d0b1f2a4ef68 \n",
      "\n",
      "=============================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "-------------------------------------------\n",
      " Successfully finished adding subscription \n",
      "-------------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'f1a960ef-b474-43f9-8621-d0b1f2a4ef68'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subscription_details = wos_client.subscriptions.add(\n",
    "        background_mode=False,\n",
    "        data_mart_id=data_mart_id,\n",
    "        service_provider_id=service_provider_id,\n",
    "        asset=Asset(\n",
    "            asset_id=model_asset_details_from_deployment[\"entity\"][\"asset\"][\"asset_id\"],\n",
    "            name=model_asset_details_from_deployment[\"entity\"][\"asset\"][\"name\"],\n",
    "            url=model_asset_details_from_deployment[\"entity\"][\"asset\"][\"url\"],\n",
    "            asset_type=AssetTypes.MODEL,\n",
    "            input_data_type=InputDataType.STRUCTURED,\n",
    "            problem_type=ProblemType.BINARY_CLASSIFICATION\n",
    "        ),\n",
    "        deployment=AssetDeploymentRequest(\n",
    "            deployment_id=asset_deployment_details['metadata']['guid'],\n",
    "            name=asset_deployment_details['entity']['name'],\n",
    "            deployment_type= DeploymentTypes.ONLINE,\n",
    "            url=asset_deployment_details['metadata']['url'],\n",
    "            scoring_endpoint=ScoringEndpointRequest(url=scoring_url) # score model without shadow deployment\n",
    "        ),\n",
    "        asset_properties=AssetPropertiesRequest(\n",
    "            label_column='Risk',\n",
    "            probability_fields=['probability'],\n",
    "            prediction_field='predictedLabel',\n",
    "            feature_fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"],\n",
    "            categorical_fields = [\"CheckingStatus\",\"CreditHistory\",\"LoanPurpose\",\"ExistingSavings\",\"EmploymentDuration\",\"Sex\",\"OthersOnLoan\",\"OwnsProperty\",\"InstallmentPlans\",\"Housing\",\"Job\",\"Telephone\",\"ForeignWorker\"],\n",
    "            training_data_reference=TrainingDataReference(type='cos',\n",
    "                                                          location=COSTrainingDataReferenceLocation(bucket = BUCKET_NAME,\n",
    "                                                                                                    file_name = training_data_file_name),\n",
    "                                                          connection=COSTrainingDataReferenceConnection.from_dict({\n",
    "                                                                        \"resource_instance_id\": COS_RESOURCE_CRN,\n",
    "                                                                        \"url\": COS_ENDPOINT,\n",
    "                                                                        \"api_key\": COS_API_KEY_ID,\n",
    "                                                                        \"iam_url\": IAM_URL}))\n",
    "        )\n",
    "    ).result\n",
    "subscription_id = subscription_details.metadata.id\n",
    "subscription_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Payload data set id:  83433d8c-302f-4b6e-a1cb-0dc4b0745e90\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "time.sleep(5)\n",
    "payload_data_set_id = None\n",
    "payload_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, \n",
    "                                                target_target_id=subscription_id, \n",
    "                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id\n",
    "if payload_data_set_id is None:\n",
    "    print(\"Payload data set not found. Please check subscription status.\")\n",
    "else:\n",
    "    print(\"Payload data set id: \", payload_data_set_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Data sets</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>data_mart_id</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>target_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>type</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>model_health</td><td style='border: 1px solid #dddddd'>2024-07-16 06:11:34.570000+00:00</td><td style='border: 1px solid #dddddd'>87eea671-1649-446b-b8e9-54bd1ec9be22</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>payload_logging_error</td><td style='border: 1px solid #dddddd'>2024-07-16 06:11:34.305000+00:00</td><td style='border: 1px solid #dddddd'>8c9c3609-08ab-4b35-bf1e-10806b80c529</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>manual_labeling</td><td style='border: 1px solid #dddddd'>2024-07-16 06:11:34.024000+00:00</td><td style='border: 1px solid #dddddd'>d5f0d283-ada2-4302-99f5-6d81c1db6d8d</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>payload_logging</td><td style='border: 1px solid #dddddd'>2024-07-16 06:11:33.622000+00:00</td><td style='border: 1px solid #dddddd'>83433d8c-302f-4b6e-a1cb-0dc4b0745e90</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>explanations</td><td style='border: 1px solid #dddddd'>2024-07-15 11:22:39.237000+00:00</td><td style='border: 1px solid #dddddd'>79002cb5-05e0-46cd-b51e-80c93d35590d</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>model_health</td><td style='border: 1px solid #dddddd'>2024-07-15 11:20:23.058000+00:00</td><td style='border: 1px solid #dddddd'>0551407f-36c0-4c15-b63e-e980d3e36c79</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>feedback</td><td style='border: 1px solid #dddddd'>2024-07-15 11:21:08.826000+00:00</td><td style='border: 1px solid #dddddd'>e63f384a-7afb-4fb8-89ff-9327d1a67aae</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>payload_logging</td><td style='border: 1px solid #dddddd'>2024-07-15 11:20:21.418000+00:00</td><td style='border: 1px solid #dddddd'>b5f897f0-20ec-46cb-9289-9e92c188603f</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>payload_logging_error</td><td style='border: 1px solid #dddddd'>2024-07-15 11:20:21.915000+00:00</td><td style='border: 1px solid #dddddd'>98e43004-9842-4f70-8fb2-fd0d52e5d136</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>manual_labeling</td><td style='border: 1px solid #dddddd'>2024-07-15 11:20:21.667000+00:00</td><td style='border: 1px solid #dddddd'>4eabe539-4f37-41f1-afa0-ddf1bed80bda</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: First 10 records were displayed.\n"
     ]
    }
   ],
   "source": [
    "wos_client.data_sets.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get subscription list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Subscriptions</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>asset_id</th><th style='border: 1px solid #dddddd'>asset_type</th><th style='border: 1px solid #dddddd'>asset_name</th><th style='border: 1px solid #dddddd'>data_mart_id</th><th style='border: 1px solid #dddddd'>deployment_id</th><th style='border: 1px solid #dddddd'>deployment_name</th><th style='border: 1px solid #dddddd'>service_provider_id</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>ce0b71a4-ecee-4e0d-b51a-123e0fad4acd</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Spark German Risk Model - Final</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>5b644de6-5b83-4930-ac85-c600b2c0ac4f</td><td style='border: 1px solid #dddddd'>Spark German Risk Deployment - Training stats</td><td style='border: 1px solid #dddddd'>6a38b72c-4aae-4fb0-b786-9b9edacf64bf</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-16 06:11:32.368000+00:00</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>085460ef94636166aea5800e9ea26168</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>GermanCreditRisk.2019.1.9.10.41.58.611</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>GermanCreditRisk.2019.1.9.10.41.58.611</td><td style='border: 1px solid #dddddd'>db2618f1-ef2b-4251-a22c-bd347702672d</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-15 11:20:20.209000+00:00</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td></tr><tr><td style='border: 1px solid #dddddd'>98783e76b0aedefc06c70bd74fa2356e</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Credit-risk-linear-learner-2021-06-24-05-32</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>cc7acf711b19c689d67e7b2a01b88ca2</td><td style='border: 1px solid #dddddd'>Credit-risk-endpoint-scoring-2021-06-24-05-32</td><td style='border: 1px solid #dddddd'>4fa5625b-5401-4a31-a423-d480ea184cda</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-11 09:30:10.069000+00:00</td><td style='border: 1px solid #dddddd'>95e480af-2fec-43ef-b34e-b3a24572b35d</td></tr><tr><td style='border: 1px solid #dddddd'>d4259f6b-e5a1-4a33-a862-d2788e5289ef</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>[asset] GCR Headless Subscription</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>aa8bd973-d680-496b-923d-f9bcbcc0bc7a</td><td style='border: 1px solid #dddddd'>[asset] GCR Headless Subscription</td><td style='border: 1px solid #dddddd'>0bf0f943-24cf-487e-b971-4a6f586effc0</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-05 13:57:01.523000+00:00</td><td style='border: 1px solid #dddddd'>345f4646-871a-4e0c-b30f-abcf9a83ea71</td></tr><tr><td style='border: 1px solid #dddddd'>ad45394c-0cfb-403c-81f6-a20f44cded7a</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Spark German Risk Model - Final</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>8db12c5c-94cf-486e-937a-6f17dadf8a56</td><td style='border: 1px solid #dddddd'>New - Spark German Risk Deployment - Final</td><td style='border: 1px solid #dddddd'>e13bfbbb-e639-4875-bbd5-6d2de8e834d0</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-04 09:15:28.269000+00:00</td><td style='border: 1px solid #dddddd'>59622413-a37f-4e76-a4d5-46967f3cf12b</td></tr><tr><td style='border: 1px solid #dddddd'>43870a88-5fb5-425a-9099-078b0469dd2a</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Amazon Mobile Review Classifier</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>8edfc920-a4bc-47cd-83fd-9e9674476dec</td><td style='border: 1px solid #dddddd'>Amazon Mobile Review Classifier deployment2</td><td style='border: 1px solid #dddddd'>84265de0-8f4b-4257-8743-7c2525e86b00</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-06-06 06:04:06.732000+00:00</td><td style='border: 1px solid #dddddd'>7828e5f6-9cc7-4cc6-b20b-7bf844c38ccc</td></tr><tr><td style='border: 1px solid #dddddd'>699a18fc-1d52-4a1b-9a25-87d9362cd3dc</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Scikit German Risk Model WML V4</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>2de6645d-c6a9-42ae-a884-4e2bb646ad33</td><td style='border: 1px solid #dddddd'>Scikit German Risk Deployment WML V4-1</td><td style='border: 1px solid #dddddd'>e13bfbbb-e639-4875-bbd5-6d2de8e834d0</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-06-02 08:26:51.212000+00:00</td><td style='border: 1px solid #dddddd'>91a07f4c-50b6-426d-93da-94438a7edc9a</td></tr><tr><td style='border: 1px solid #dddddd'>a7fc7ed9-f000-4f27-988a-54fbc8e397e6</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>GermanCreditRiskModelChallenger</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>38059feb-e4c5-4037-90ea-a18f77a8ec5b</td><td style='border: 1px solid #dddddd'>GermanCreditRiskModelChallenger</td><td style='border: 1px solid #dddddd'>19fc94cc-9ed2-4ac6-a98e-1acf1a9e729e</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-05-16 06:09:55.832000+00:00</td><td style='border: 1px solid #dddddd'>81044f9c-27a8-42f2-90be-9cef8b33fe39</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wos_client.subscriptions.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Score the model so we can configure monitors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the WML service has been bound and the subscription has been created, we need to send a request to the model before we configure OpenScale. This allows OpenScale to create a payload log in the datamart with the correct schema, so it can capture data coming into and out of the model. First, the code gets the model deployment's endpoint URL, and then sends a few records for predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Single record scoring result: \n",
      " fields: ['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker', 'CheckingStatus_IX', 'CreditHistory_IX', 'LoanPurpose_IX', 'ExistingSavings_IX', 'EmploymentDuration_IX', 'Sex_IX', 'OthersOnLoan_IX', 'OwnsProperty_IX', 'InstallmentPlans_IX', 'Housing_IX', 'Job_IX', 'Telephone_IX', 'ForeignWorker_IX', 'features', 'rawPrediction', 'probability', 'prediction', 'predictedLabel'] \n",
      " values:  ['no_checking', 13, 'credits_paid_to_date', 'car_new', 1343, '100_to_500', '1_to_4', 2, 'female', 'none', 3, 'savings_insurance', 46, 'none', 'own', 2, 'skilled', 1, 'none', 'yes', 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, [20, [1, 3, 5, 13, 14, 15, 16, 17, 18, 19], [1.0, 1.0, 1.0, 13.0, 1343.0, 2.0, 3.0, 46.0, 2.0, 1.0]], [12.979298576936431, 7.0207014230635725], [0.6489649288468214, 0.35103507115317856], 0.0, 'No Risk']\n"
     ]
    }
   ],
   "source": [
    "fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\n",
    "values = [\n",
    "  [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n",
    "  [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n",
    "]\n",
    "\n",
    "payload_scoring = {\"input_data\": [{\"fields\": fields, \"values\": values}]}\n",
    "predictions = wml_client.deployments.score(deployment_uid, payload_scoring)\n",
    "\n",
    "print(\"Single record scoring result:\", \"\\n fields:\", predictions[\"predictions\"][0][\"fields\"], \"\\n values: \", predictions[\"predictions\"][0][\"values\"][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check if WML payload logging worked else manually store payload records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of records in the payload logging table: 0\n",
      "Payload logging did not happen, performing explicit payload logging.\n",
      "\n",
      "\n",
      "==========================================================================================\n",
      "\n",
      " Waiting for end of storing records with request id: d0131b2e-114f-449e-957c-03724c97f66f \n",
      "\n",
      "==========================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Successfully finished storing records \n",
      "---------------------------------------\n",
      "\n",
      "\n",
      "Number of records in the payload logging table: 8\n"
     ]
    }
   ],
   "source": [
    "import uuid\n",
    "from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord\n",
    "time.sleep(5)\n",
    "pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)\n",
    "print(\"Number of records in the payload logging table: {}\".format(pl_records_count))\n",
    "if pl_records_count == 0:\n",
    "    print(\"Payload logging did not happen, performing explicit payload logging.\")\n",
    "    wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(\n",
    "                   scoring_id=str(uuid.uuid4()),\n",
    "                   request=payload_scoring,\n",
    "                   response={\"fields\": predictions['predictions'][0]['fields'], \"values\":predictions['predictions'][0]['values']},\n",
    "                   response_time=460\n",
    "               )],background_mode=False)\n",
    "    time.sleep(5)\n",
    "    pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)\n",
    "    print(\"Number of records in the payload logging table: {}\".format(pl_records_count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quality monitoring and feedback logging <a name=\"quality\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enable quality monitoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below waits ten seconds to allow the payload logging table to be set up before it begins enabling monitors. First, it turns on the quality (accuracy) monitor and sets an alert threshold of 70%. OpenScale will show an alert on the dashboard if the model accuracy measurement (area under the curve, in the case of a binary classifier) falls below this threshold.\n",
    "\n",
    "The second paramater supplied, min_records, specifies the minimum number of feedback records OpenScale needs before it calculates a new measurement. The quality monitor runs hourly, but the accuracy reading in the dashboard will not change until an additional 50 feedback records have been added, via the user interface, the Python client, or the supplied feedback endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation f16dff88-1305-4584-8c7d-3056728e79a4 \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "preparing\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "time.sleep(10)\n",
    "target = Target(\n",
    "        target_type=TargetTypes.SUBSCRIPTION,\n",
    "        target_id=subscription_id\n",
    ")\n",
    "parameters = {\n",
    "    \"min_feedback_data_size\": 50\n",
    "}\n",
    "thresholds = [\n",
    "                {\n",
    "                    \"metric_id\": \"area_under_roc\",\n",
    "                    \"type\": \"lower_limit\",\n",
    "                    \"value\": .80\n",
    "                }\n",
    "            ]\n",
    "quality_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.QUALITY.ID,\n",
    "    target=target,\n",
    "    parameters=parameters,\n",
    "    thresholds=thresholds\n",
    ").result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'f16dff88-1305-4584-8c7d-3056728e79a4'"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quality_monitor_instance_id = quality_monitor_details.metadata.id\n",
    "quality_monitor_instance_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feedback logging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below downloads and stores enough feedback data to meet the minimum threshold so that OpenScale can calculate a new accuracy measurement. It then kicks off the accuracy monitor. The monitors run hourly, or can be initiated via the Python API, the REST API, or the graphical user interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm: additional_feedback_data_v2.json: No such file or directory\n",
      "--2024-07-16 11:42:41--  https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/additional_feedback_data_v2.json\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8003::154, 2606:50c0:8000::154, 2606:50c0:8001::154, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8003::154|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 50890 (50K) [text/plain]\n",
      "Saving to: ‘additional_feedback_data_v2.json’\n",
      "\n",
      "additional_feedback 100%[===================>]  49.70K  --.-KB/s    in 0.03s   \n",
      "\n",
      "2024-07-16 11:42:42 (1.92 MB/s) - ‘additional_feedback_data_v2.json’ saved [50890/50890]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm additional_feedback_data_v2.json\n",
    "!wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/additional_feedback_data_v2.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get feedback logging dataset ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"data_sets\": [\n",
      "    {\n",
      "      \"metadata\": {\n",
      "        \"id\": \"3b44db5f-e7d0-4e57-a4b2-8d6622b23be7\",\n",
      "        \"crn\": \"crn:v1:bluemix:public:aiopenscale:us-south:a/181ed6cc388f47bd9d862fe066f9cfce:80e6093f-5acf-4eb7-9da6-7ba9bf56a929:data_set:3b44db5f-e7d0-4e57-a4b2-8d6622b23be7\",\n",
      "        \"url\": \"/v2/data_sets/3b44db5f-e7d0-4e57-a4b2-8d6622b23be7\",\n",
      "        \"created_at\": \"2024-07-16T06:12:30.424000Z\",\n",
      "        \"created_by\": \"iam-ServiceId-2e5c9fda-38bf-4279-9712-cdb3b6f3a7ad\",\n",
      "        \"modified_at\": \"2024-07-16T06:12:30.693000Z\",\n",
      "        \"modified_by\": \"iam-ServiceId-2e5c9fda-38bf-4279-9712-cdb3b6f3a7ad\"\n",
      "      },\n",
      "      \"entity\": {\n",
      "        \"data_mart_id\": \"80e6093f-5acf-4eb7-9da6-7ba9bf56a929\",\n",
      "        \"name\": \"f1a960ef-b474-43f9-8621-d0b1f2a4ef68_feedback\",\n",
      "        \"description\": \"f1a960ef-b474-43f9-8621-d0b1f2a4ef68_feedback\",\n",
      "        \"type\": \"feedback\",\n",
      "        \"target\": {\n",
      "          \"target_type\": \"subscription\",\n",
      "          \"target_id\": \"f1a960ef-b474-43f9-8621-d0b1f2a4ef68\"\n",
      "        },\n",
      "        \"schema_update_mode\": \"auto\",\n",
      "        \"data_schema\": {\n",
      "          \"type\": \"struct\",\n",
      "          \"fields\": [\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"columnInfo\": {\n",
      "                  \"columnLength\": 64\n",
      "                },\n",
      "                \"measure\": \"discrete\",\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"CheckingStatus\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"LoanDuration\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"columnInfo\": {\n",
      "                  \"columnLength\": 64\n",
      "                },\n",
      "                \"measure\": \"discrete\",\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"CreditHistory\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"columnInfo\": {\n",
      "                  \"columnLength\": 64\n",
      "                },\n",
      "                \"measure\": \"discrete\",\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"LoanPurpose\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"LoanAmount\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"columnInfo\": {\n",
      "                  \"columnLength\": 64\n",
      "                },\n",
      "                \"measure\": \"discrete\",\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"ExistingSavings\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"columnInfo\": {\n",
      "                  \"columnLength\": 64\n",
      "                },\n",
      "                \"measure\": \"discrete\",\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"EmploymentDuration\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"InstallmentPercent\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"columnInfo\": {\n",
      "                  \"columnLength\": 64\n",
      "                },\n",
      "                \"measure\": \"discrete\",\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"Sex\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"columnInfo\": {\n",
      "                  \"columnLength\": 64\n",
      "                },\n",
      "                \"measure\": \"discrete\",\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"OthersOnLoan\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"CurrentResidenceDuration\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"columnInfo\": {\n",
      "                  \"columnLength\": 64\n",
      "                },\n",
      "                \"measure\": \"discrete\",\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"OwnsProperty\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"Age\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"columnInfo\": {\n",
      "                  \"columnLength\": 64\n",
      "                },\n",
      "                \"measure\": \"discrete\",\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"InstallmentPlans\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"columnInfo\": {\n",
      "                  \"columnLength\": 64\n",
      "                },\n",
      "                \"measure\": \"discrete\",\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"Housing\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"ExistingCreditsCount\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"columnInfo\": {\n",
      "                  \"columnLength\": 64\n",
      "                },\n",
      "                \"measure\": \"discrete\",\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"Job\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"Dependents\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"columnInfo\": {\n",
      "                  \"columnLength\": 64\n",
      "                },\n",
      "                \"measure\": \"discrete\",\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"Telephone\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"columnInfo\": {\n",
      "                  \"columnLength\": 64\n",
      "                },\n",
      "                \"measure\": \"discrete\",\n",
      "                \"modeling_role\": \"feature\"\n",
      "              },\n",
      "              \"name\": \"ForeignWorker\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"columnInfo\": {\n",
      "                  \"columnLength\": 64\n",
      "                },\n",
      "                \"modeling_role\": \"target\"\n",
      "              },\n",
      "              \"name\": \"Risk\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"modeling_role\": \"record-id\",\n",
      "                \"primary_key\": true\n",
      "              },\n",
      "              \"name\": \"record_id\",\n",
      "              \"nullable\": false,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"modeling_role\": \"record-timestamp\"\n",
      "              },\n",
      "              \"name\": \"record_timestamp\",\n",
      "              \"nullable\": false,\n",
      "              \"type\": \"timestamp\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {\n",
      "                \"transaction_id_key\": true\n",
      "              },\n",
      "              \"name\": \"transaction_id\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {},\n",
      "              \"name\": \"_original_prediction\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {},\n",
      "              \"name\": \"_original_probability\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": {\n",
      "                \"containsNull\": true,\n",
      "                \"elementType\": \"double\",\n",
      "                \"type\": \"array\"\n",
      "              }\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {},\n",
      "              \"name\": \"_debiased_prediction\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"metadata\": {},\n",
      "              \"name\": \"_debiased_probability\",\n",
      "              \"nullable\": true,\n",
      "              \"type\": {\n",
      "                \"containsNull\": true,\n",
      "                \"elementType\": \"double\",\n",
      "                \"type\": \"array\"\n",
      "              }\n",
      "            }\n",
      "          ]\n",
      "        },\n",
      "        \"location\": {\n",
      "          \"table_name\": \"80e6093f-5acf-4eb7-9da6-7ba9bf56a929.Feedback_f1a960ef-b474-43f9-8621-d0b1f2a4ef68\"\n",
      "        },\n",
      "        \"status\": {\n",
      "          \"state\": \"active\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "feedback_dataset_id = None\n",
    "feedback_dataset = wos_client.data_sets.list(type=DataSetTypes.FEEDBACK, \n",
    "                                                target_target_id=subscription_id, \n",
    "                                                target_target_type=TargetTypes.SUBSCRIPTION).result\n",
    "print(feedback_dataset)\n",
    "feedback_dataset_id = feedback_dataset.data_sets[0].metadata.id\n",
    "if feedback_dataset_id is None:\n",
    "    print(\"Feedback data set not found. Please check quality monitor status.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('additional_feedback_data_v2.json') as feedback_file:\n",
    "    additional_feedback_data = json.load(feedback_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "==========================================================================================\n",
      "\n",
      " Waiting for end of storing records with request id: 7a62e26f-1e79-4253-bf9c-857e1b54d00b \n",
      "\n",
      "==========================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Successfully finished storing records \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<ibm_cloud_sdk_core.detailed_response.DetailedResponse at 0x1481bbf10>"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time.sleep(20) # Sleep to make sure that feedback dataset id is in active state\n",
    "wos_client.data_sets.store_records(feedback_dataset_id, request_body=additional_feedback_data, background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "98"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wos_client.data_sets.get_records_count(data_set_id=feedback_dataset_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "========================================================================\n",
      "\n",
      " Waiting for end of monitoring run 71790afc-05d2-4f4b-be08-f0afc85da02d \n",
      "\n",
      "========================================================================\n",
      "\n",
      "\n",
      "\n",
      "finished\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "run_details = wos_client.monitor_instances.run(monitor_instance_id=quality_monitor_instance_id, background_mode=False).result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>f16dff88-1305-4584-8c7d-3056728e79a4 Monitor Runs Metrics from: 2024-07-09 11:43:57.801351  till: 2024-07-16 11:43:57.801360</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower_limit</th><th style='border: 1px solid #dddddd'>upper_limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>monitor_instance_id</th><th style='border: 1px solid #dddddd'>run_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>target_id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2024-07-16 06:13:19.969000+00:00</td><td style='border: 1px solid #dddddd'>true_positive_rate</td><td style='border: 1px solid #dddddd'>1267b942-5fe0-423c-8263-b32816c30099</td><td style='border: 1px solid #dddddd'>0.36363636363636365</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>f16dff88-1305-4584-8c7d-3056728e79a4</td><td style='border: 1px solid #dddddd'>92b130ed-fc31-43c2-8b0e-a5f8d3a91345</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:13:19.969000+00:00</td><td style='border: 1px solid #dddddd'>area_under_roc</td><td style='border: 1px solid #dddddd'>1267b942-5fe0-423c-8263-b32816c30099</td><td style='border: 1px solid #dddddd'>0.6587412587412588</td><td style='border: 1px solid #dddddd'>0.8</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>f16dff88-1305-4584-8c7d-3056728e79a4</td><td style='border: 1px solid #dddddd'>92b130ed-fc31-43c2-8b0e-a5f8d3a91345</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:13:19.969000+00:00</td><td style='border: 1px solid #dddddd'>precision</td><td style='border: 1px solid #dddddd'>1267b942-5fe0-423c-8263-b32816c30099</td><td style='border: 1px solid #dddddd'>0.8</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>f16dff88-1305-4584-8c7d-3056728e79a4</td><td style='border: 1px solid #dddddd'>92b130ed-fc31-43c2-8b0e-a5f8d3a91345</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:13:19.969000+00:00</td><td style='border: 1px solid #dddddd'>matthews_correlation_coefficient</td><td style='border: 1px solid #dddddd'>1267b942-5fe0-423c-8263-b32816c30099</td><td style='border: 1px solid #dddddd'>0.41672426371926674</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>f16dff88-1305-4584-8c7d-3056728e79a4</td><td style='border: 1px solid #dddddd'>92b130ed-fc31-43c2-8b0e-a5f8d3a91345</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:13:19.969000+00:00</td><td style='border: 1px solid #dddddd'>f1_measure</td><td style='border: 1px solid #dddddd'>1267b942-5fe0-423c-8263-b32816c30099</td><td style='border: 1px solid #dddddd'>0.5000000000000001</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>f16dff88-1305-4584-8c7d-3056728e79a4</td><td style='border: 1px solid #dddddd'>92b130ed-fc31-43c2-8b0e-a5f8d3a91345</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:13:19.969000+00:00</td><td style='border: 1px solid #dddddd'>accuracy</td><td style='border: 1px solid #dddddd'>1267b942-5fe0-423c-8263-b32816c30099</td><td style='border: 1px solid #dddddd'>0.7551020408163265</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>f16dff88-1305-4584-8c7d-3056728e79a4</td><td style='border: 1px solid #dddddd'>92b130ed-fc31-43c2-8b0e-a5f8d3a91345</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:13:19.969000+00:00</td><td style='border: 1px solid #dddddd'>label_skew</td><td style='border: 1px solid #dddddd'>1267b942-5fe0-423c-8263-b32816c30099</td><td style='border: 1px solid #dddddd'>0.6909336273400493</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>f16dff88-1305-4584-8c7d-3056728e79a4</td><td style='border: 1px solid #dddddd'>92b130ed-fc31-43c2-8b0e-a5f8d3a91345</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:13:19.969000+00:00</td><td style='border: 1px solid #dddddd'>gini_coefficient</td><td style='border: 1px solid #dddddd'>1267b942-5fe0-423c-8263-b32816c30099</td><td style='border: 1px solid #dddddd'>0.31748251748251755</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>f16dff88-1305-4584-8c7d-3056728e79a4</td><td style='border: 1px solid #dddddd'>92b130ed-fc31-43c2-8b0e-a5f8d3a91345</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:13:19.969000+00:00</td><td style='border: 1px solid #dddddd'>log_loss</td><td style='border: 1px solid #dddddd'>1267b942-5fe0-423c-8263-b32816c30099</td><td style='border: 1px solid #dddddd'>0.4493805793027406</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>f16dff88-1305-4584-8c7d-3056728e79a4</td><td style='border: 1px solid #dddddd'>92b130ed-fc31-43c2-8b0e-a5f8d3a91345</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:13:19.969000+00:00</td><td style='border: 1px solid #dddddd'>false_positive_rate</td><td style='border: 1px solid #dddddd'>1267b942-5fe0-423c-8263-b32816c30099</td><td style='border: 1px solid #dddddd'>0.046153846153846156</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>f16dff88-1305-4584-8c7d-3056728e79a4</td><td style='border: 1px solid #dddddd'>92b130ed-fc31-43c2-8b0e-a5f8d3a91345</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: First 10 records were displayed.\n"
     ]
    }
   ],
   "source": [
    "wos_client.monitor_instances.show_metrics(monitor_instance_id=quality_monitor_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fairness, drift monitoring and explanations <a name=\"fairness\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fairness configuration\n",
    "\n",
    "The code below configures fairness monitoring for our model. It turns on monitoring for two features, Sex and Age. In each case, we must specify:\n",
    "\n",
    "  * Which model feature to monitor\n",
    "  * One or more **majority** groups, which are values of that feature that we expect to receive a higher percentage of favorable outcomes\n",
    "  * One or more **minority** groups, which are values of that feature that we expect to receive a higher percentage of unfavorable outcomes\n",
    "  * The threshold at which we would like OpenScale to display an alert if the fairness measurement falls below (in this case, 95%)\n",
    "\n",
    "Additionally, we must specify which outcomes from the model are favourable outcomes, and which are unfavourable. We must also provide the number of records OpenScale will use to calculate the fairness score. In this case, OpenScale's fairness monitor will run hourly, but will not calculate a new fairness rating until at least 200 records have been added. Finally, to calculate fairness, OpenScale must perform some calculations on the training data, so we provide the dataframe containing the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Monitor instances</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>data_mart_id</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>target_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>model_health</td><td style='border: 1px solid #dddddd'>2024-07-16 06:11:34.218000+00:00</td><td style='border: 1px solid #dddddd'>bb904906-53a8-425b-8831-d6f8b499b18a</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>2024-07-16 06:12:29.680000+00:00</td><td style='border: 1px solid #dddddd'>f16dff88-1305-4584-8c7d-3056728e79a4</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>performance</td><td style='border: 1px solid #dddddd'>2024-07-16 06:11:36.282000+00:00</td><td style='border: 1px solid #dddddd'>ab36daa4-b83e-42a4-9a81-b4cb66b1b8ec</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>345f4646-871a-4e0c-b30f-abcf9a83ea71</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>2024-07-05 14:08:34.802000+00:00</td><td style='border: 1px solid #dddddd'>2f9e942b-5170-419f-989c-bc2c7e1c923b</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>7828e5f6-9cc7-4cc6-b20b-7bf844c38ccc</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>model_health</td><td style='border: 1px solid #dddddd'>2024-06-06 06:04:08.111000+00:00</td><td style='border: 1px solid #dddddd'>75d15cfc-3ded-41ff-b1bf-805ce134e6be</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>91a07f4c-50b6-426d-93da-94438a7edc9a</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>2024-07-02 10:04:06.083000+00:00</td><td style='border: 1px solid #dddddd'>e130d4dc-5b57-4808-8cee-1f834b33f3e4</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>345f4646-871a-4e0c-b30f-abcf9a83ea71</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>model_health</td><td style='border: 1px solid #dddddd'>2024-07-05 13:57:02.336000+00:00</td><td style='border: 1px solid #dddddd'>37464a51-58b9-4620-afe6-d710be384daa</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>drift</td><td style='border: 1px solid #dddddd'>2024-07-15 11:31:15.872000+00:00</td><td style='border: 1px solid #dddddd'>dc52c272-b69c-47c3-abb6-c0bb4e1cfe4d</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>95e480af-2fec-43ef-b34e-b3a24572b35d</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>model_health</td><td style='border: 1px solid #dddddd'>2024-07-11 09:30:12.810000+00:00</td><td style='border: 1px solid #dddddd'>b53585b2-a86b-4d53-b4ec-7e3be95e1739</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>91a07f4c-50b6-426d-93da-94438a7edc9a</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>model_health</td><td style='border: 1px solid #dddddd'>2024-06-02 08:26:53.650000+00:00</td><td style='border: 1px solid #dddddd'>2c69e8d5-09bc-4c69-8e98-e63ff5a5098d</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: First 10 records were displayed.\n"
     ]
    }
   ],
   "source": [
    "wos_client.monitor_instances.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation 3c7dfc4b-d9c8-4802-a03d-ba91b92d5256 \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'3c7dfc4b-d9c8-4802-a03d-ba91b92d5256'"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "    \"features\": [\n",
    "        {\"feature\": \"Sex\",\n",
    "         \"majority\": ['male'],\n",
    "         \"minority\": ['female'],\n",
    "         \"threshold\": 0.95\n",
    "         },\n",
    "        {\"feature\": \"Age\",\n",
    "         \"majority\": [[26, 75]],\n",
    "         \"minority\": [[18, 25]],\n",
    "         \"threshold\": 0.95\n",
    "         }\n",
    "    ],\n",
    "    \"favourable_class\": [\"No Risk\"],\n",
    "    \"unfavourable_class\": [\"Risk\"],\n",
    "    \"min_records\": 100\n",
    "}\n",
    "\n",
    "fairness_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.FAIRNESS.ID,\n",
    "    target=target,\n",
    "    parameters=parameters).result\n",
    "fairness_monitor_instance_id =fairness_monitor_details.metadata.id\n",
    "fairness_monitor_instance_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drift configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Note: you can choose to enable/disable (True or False) model or data drift within config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation b9a3b5d4-16bc-400c-9e6c-0e9f593101c7 \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "preparing\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'b9a3b5d4-16bc-400c-9e6c-0e9f593101c7'"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "monitor_instances = wos_client.monitor_instances.list().result.monitor_instances\n",
    "for monitor_instance in monitor_instances:\n",
    "    monitor_def_id=monitor_instance.entity.monitor_definition_id\n",
    "    if monitor_def_id == \"drift\" and monitor_instance.entity.target.target_id == subscription_id:\n",
    "        wos_client.monitor_instances.delete(monitor_instance.metadata.id)\n",
    "        print('Deleted existing drift monitor instance with id: ', monitor_instance.metadata.id)\n",
    "\n",
    "\n",
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "    \"min_samples\": 100,\n",
    "    \"drift_threshold\": 0.1,\n",
    "    \"train_drift_model\": True,\n",
    "    \"enable_model_drift\": False,\n",
    "    \"enable_data_drift\": True\n",
    "}\n",
    "\n",
    "drift_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "drift_monitor_instance_id = drift_monitor_details.metadata.id\n",
    "drift_monitor_instance_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drift V2 Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "monitor_instances = wos_client.monitor_instances.list().result.monitor_instances\n",
    "for monitor_instance in monitor_instances:\n",
    "    monitor_def_id=monitor_instance.entity.monitor_definition_id\n",
    "    if monitor_def_id == \"drift_v2\" and monitor_instance.entity.target.target_id == subscription_id:\n",
    "        wos_client.monitor_instances.delete(monitor_instance.metadata.id)\n",
    "        print('Deleted existing drift v2 monitor instance with id: ', monitor_instance.metadata.id)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finding feature importances\n",
    "\n",
    "This is done because we will deduce the important and most important features to the model. This will help in narrowing down the analysis of Drift evaluation in the UI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_importance = dict(zip(features, model_rf.stages[-3].featureImportances))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'c61a2e92-9218-430a-96b2-0d6262fba0ae'"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=subscription_id\n",
    ")\n",
    "\n",
    "parameters = {\n",
    "        \"min_samples\": 100,\n",
    "        \"max_samples\": 1000,\n",
    "        \"train_archive\": True,\n",
    "        \"features\": {\n",
    "            \"fields\": features,\n",
    "            \"importances\": feature_importance\n",
    "        }\n",
    "    }\n",
    "drift_v2_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT_V2.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "drift_v2_monitor_instance_id = drift_v2_monitor_details.metadata.id\n",
    "drift_v2_monitor_instance_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Score the model again now that monitoring is configured"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This next section randomly selects 200 records from the data feed and sends those records to the model for predictions. This is enough to exceed the minimum threshold for records set in the previous section, which allows OpenScale to begin calculating fairness."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--  1 nelwin  staff   2.9M Jul 16 11:48 german_credit_feed.json\n"
     ]
    }
   ],
   "source": [
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/german_credit_feed.json\n",
    "!ls -lh german_credit_feed.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Score 200 randomly chosen records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Payload logging did not happen, performing explicit payload logging.\n",
      "Number of records in payload logging table:  208\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "with open('german_credit_feed.json', 'r') as scoring_file:\n",
    "    scoring_data = json.load(scoring_file)\n",
    "\n",
    "fields = scoring_data['fields']\n",
    "values = []\n",
    "for _ in range(200):\n",
    "    values.append(random.choice(scoring_data['values']))\n",
    "payload_scoring = {\"input_data\": [{\"fields\": fields, \"values\": values}]}\n",
    "\n",
    "scoring_response = wml_client.deployments.score(deployment_uid, payload_scoring)\n",
    "\n",
    "time.sleep(5)\n",
    "pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)\n",
    "\n",
    "if pl_records_count == 8:\n",
    "    print(\"Payload logging did not happen, performing explicit payload logging.\")\n",
    "    wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(\n",
    "                   scoring_id=str(uuid.uuid4()),\n",
    "                   request=payload_scoring,\n",
    "                   response={\"fields\": scoring_response['predictions'][0]['fields'], \"values\":scoring_response['predictions'][0]['values']},\n",
    "                   response_time=460\n",
    "               )])\n",
    "    time.sleep(5)\n",
    "    pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)\n",
    "    print(\"Number of records in the payload logging table: {}\".format(pl_records_count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Now in payload table should be total 208 records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of records in payload table:  208\n"
     ]
    }
   ],
   "source": [
    "print('Number of records in payload table: ', wos_client.data_sets.get_records_count(data_set_id=payload_data_set_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run fairness monitor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Kick off a fairness monitor run on current data. The monitor runs hourly, but can be manually initiated using the Python client, the REST API, or the graphical user interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "========================================================================\n",
      "\n",
      " Waiting for end of monitoring run 3c7dfc4b-d9c8-4802-a03d-ba91b92d5256 \n",
      "\n",
      "========================================================================\n",
      "\n",
      "\n",
      "\n",
      "finished\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "run_details = wos_client.monitor_instances.run(monitor_instance_id=fairness_monitor_instance_id, background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>3c7dfc4b-d9c8-4802-a03d-ba91b92d5256 Monitor Runs Metrics from: 2024-07-09 12:14:29.468891  till: 2024-07-16 12:14:29.468914</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower_limit</th><th style='border: 1px solid #dddddd'>upper_limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>monitor_instance_id</th><th style='border: 1px solid #dddddd'>run_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>target_id</th>\n",
       "                \n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "time.sleep(10)\n",
    "\n",
    "wos_client.monitor_instances.show_metrics(monitor_instance_id=fairness_monitor_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run drift monitor\n",
    "\n",
    "\n",
    "Kick off a drift monitor run on current data. The monitor runs every hour, but can be manually initiated using the Python client, the REST API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "========================================================================\n",
      "\n",
      " Waiting for end of monitoring run 31c96d8a-9d38-4695-825e-466e7265cb71 \n",
      "\n",
      "========================================================================\n",
      "\n",
      "\n",
      "\n",
      "finished\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "drift_run_details = wos_client.monitor_instances.run(monitor_instance_id=drift_monitor_instance_id, background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>b9a3b5d4-16bc-400c-9e6c-0e9f593101c7 Monitor Runs Metrics from: 2024-07-09 11:49:31.048768  till: 2024-07-16 11:49:31.048792</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower_limit</th><th style='border: 1px solid #dddddd'>upper_limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>monitor_instance_id</th><th style='border: 1px solid #dddddd'>run_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>target_id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2024-07-16 06:19:17.267002+00:00</td><td style='border: 1px solid #dddddd'>data_drift_magnitude</td><td style='border: 1px solid #dddddd'>8d86edf2-07fd-4a9f-b625-11c838d21e02</td><td style='border: 1px solid #dddddd'>0.125</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.1</td><td style='border: 1px solid #dddddd'>[]</td><td style='border: 1px solid #dddddd'>drift</td><td style='border: 1px solid #dddddd'>b9a3b5d4-16bc-400c-9e6c-0e9f593101c7</td><td style='border: 1px solid #dddddd'>31c96d8a-9d38-4695-825e-466e7265cb71</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "time.sleep(5)\n",
    "\n",
    "wos_client.monitor_instances.show_metrics(monitor_instance_id=drift_monitor_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Drift V2 monitor\n",
    "\n",
    "\n",
    "Kick off a drift v2 monitor run on current data. The monitor runs every day, but can be manually initiated using the Python client, the REST API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "========================================================================\n",
      "\n",
      " Waiting for end of monitoring run 0fd0b8bc-944d-49f0-a71d-7bf2ec7098c9 \n",
      "\n",
      "========================================================================\n",
      "\n",
      "\n",
      "\n",
      "running................................................\n",
      "finished\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "drift_v2_run_details = wos_client.monitor_instances.run(monitor_instance_id=drift_v2_monitor_instance_id, background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>c61a2e92-9218-430a-96b2-0d6262fba0ae Monitor Runs Metrics from: 2024-07-09 11:54:23.509228  till: 2024-07-16 11:54:23.509436</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower_limit</th><th style='border: 1px solid #dddddd'>upper_limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>monitor_instance_id</th><th style='border: 1px solid #dddddd'>run_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>target_id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2024-07-16 06:19:52.904904+00:00</td><td style='border: 1px solid #dddddd'>records_processed</td><td style='border: 1px solid #dddddd'>b7e32f25-be24-4583-95b3-bf0c52109ffb</td><td style='border: 1px solid #dddddd'>208.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['algorithm_used:total_variation', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for No Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>c61a2e92-9218-430a-96b2-0d6262fba0ae</td><td style='border: 1px solid #dddddd'>0fd0b8bc-944d-49f0-a71d-7bf2ec7098c9</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:19:52.904904+00:00</td><td style='border: 1px solid #dddddd'>confidence_drift_score</td><td style='border: 1px solid #dddddd'>b7e32f25-be24-4583-95b3-bf0c52109ffb</td><td style='border: 1px solid #dddddd'>0.1862</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.05</td><td style='border: 1px solid #dddddd'>['algorithm_used:total_variation', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for No Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>c61a2e92-9218-430a-96b2-0d6262fba0ae</td><td style='border: 1px solid #dddddd'>0fd0b8bc-944d-49f0-a71d-7bf2ec7098c9</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:19:52.904904+00:00</td><td style='border: 1px solid #dddddd'>records_processed</td><td style='border: 1px solid #dddddd'>b7e32f25-be24-4583-95b3-bf0c52109ffb</td><td style='border: 1px solid #dddddd'>208.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['algorithm_used:overlap_coefficient', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for No Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>c61a2e92-9218-430a-96b2-0d6262fba0ae</td><td style='border: 1px solid #dddddd'>0fd0b8bc-944d-49f0-a71d-7bf2ec7098c9</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:19:52.904904+00:00</td><td style='border: 1px solid #dddddd'>confidence_drift_score</td><td style='border: 1px solid #dddddd'>b7e32f25-be24-4583-95b3-bf0c52109ffb</td><td style='border: 1px solid #dddddd'>0.1868</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.05</td><td style='border: 1px solid #dddddd'>['algorithm_used:overlap_coefficient', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for No Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>c61a2e92-9218-430a-96b2-0d6262fba0ae</td><td style='border: 1px solid #dddddd'>0fd0b8bc-944d-49f0-a71d-7bf2ec7098c9</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:19:58.188821+00:00</td><td style='border: 1px solid #dddddd'>records_processed</td><td style='border: 1px solid #dddddd'>e314eb41-daf5-4bc4-bf92-c56cccba4eb7</td><td style='border: 1px solid #dddddd'>208.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['algorithm_used:total_variation', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>c61a2e92-9218-430a-96b2-0d6262fba0ae</td><td style='border: 1px solid #dddddd'>0fd0b8bc-944d-49f0-a71d-7bf2ec7098c9</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:19:58.188821+00:00</td><td style='border: 1px solid #dddddd'>confidence_drift_score</td><td style='border: 1px solid #dddddd'>e314eb41-daf5-4bc4-bf92-c56cccba4eb7</td><td style='border: 1px solid #dddddd'>0.1657</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.05</td><td style='border: 1px solid #dddddd'>['algorithm_used:total_variation', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>c61a2e92-9218-430a-96b2-0d6262fba0ae</td><td style='border: 1px solid #dddddd'>0fd0b8bc-944d-49f0-a71d-7bf2ec7098c9</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:19:58.188821+00:00</td><td style='border: 1px solid #dddddd'>records_processed</td><td style='border: 1px solid #dddddd'>e314eb41-daf5-4bc4-bf92-c56cccba4eb7</td><td style='border: 1px solid #dddddd'>208.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['algorithm_used:overlap_coefficient', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>c61a2e92-9218-430a-96b2-0d6262fba0ae</td><td style='border: 1px solid #dddddd'>0fd0b8bc-944d-49f0-a71d-7bf2ec7098c9</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:19:58.188821+00:00</td><td style='border: 1px solid #dddddd'>confidence_drift_score</td><td style='border: 1px solid #dddddd'>e314eb41-daf5-4bc4-bf92-c56cccba4eb7</td><td style='border: 1px solid #dddddd'>0.1667</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.05</td><td style='border: 1px solid #dddddd'>['algorithm_used:overlap_coefficient', 'computed_on:payload', 'field_type:class', 'field_name:Class Probability for Risk']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>c61a2e92-9218-430a-96b2-0d6262fba0ae</td><td style='border: 1px solid #dddddd'>0fd0b8bc-944d-49f0-a71d-7bf2ec7098c9</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:19:58.596595+00:00</td><td style='border: 1px solid #dddddd'>records_processed</td><td style='border: 1px solid #dddddd'>75c7061d-8b95-4f94-bfc7-9c4b8d354476</td><td style='border: 1px solid #dddddd'>208.0</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['algorithm_used:jensen_shannon', 'computed_on:payload', 'field_type:class', 'field_name:predictedLabel']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>c61a2e92-9218-430a-96b2-0d6262fba0ae</td><td style='border: 1px solid #dddddd'>0fd0b8bc-944d-49f0-a71d-7bf2ec7098c9</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-16 06:19:58.596595+00:00</td><td style='border: 1px solid #dddddd'>prediction_drift_score</td><td style='border: 1px solid #dddddd'>75c7061d-8b95-4f94-bfc7-9c4b8d354476</td><td style='border: 1px solid #dddddd'>0.0469</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.05</td><td style='border: 1px solid #dddddd'>['algorithm_used:jensen_shannon', 'computed_on:payload', 'field_type:class', 'field_name:predictedLabel']</td><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>c61a2e92-9218-430a-96b2-0d6262fba0ae</td><td style='border: 1px solid #dddddd'>0fd0b8bc-944d-49f0-a71d-7bf2ec7098c9</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>f1a960ef-b474-43f9-8621-d0b1f2a4ef68</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: First 10 records were displayed.\n"
     ]
    }
   ],
   "source": [
    "time.sleep(15)\n",
    "\n",
    "wos_client.monitor_instances.show_metrics(monitor_instance_id=drift_v2_monitor_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure Explainability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we provide OpenScale with the training data to enable and configure the explainability features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation 7a3a484f-20b8-45e7-ad3c-ee1b02d760cf \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "preparing..\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=subscription_id\n",
    ")\n",
    "parameters = {\n",
    "    \"enabled\": True\n",
    "}\n",
    "explainability_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.EXPLAINABILITY.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "explainability_monitor_id = explainability_details.metadata.id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run explanation for sample record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running explanations on scoring IDs: ['7512f474-6c88-44bd-a591-24350774b1ad-1']\n",
      "{\n",
      "  \"metadata\": {\n",
      "    \"explanation_task_ids\": [\n",
      "      \"2377ff91-77c7-441a-b7c8-b70a48ab0a59\"\n",
      "    ],\n",
      "    \"created_by\": \"IBMid-662005298W\",\n",
      "    \"created_at\": \"2024-07-16T06:24:59.486239Z\"\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "pl_records_resp = wos_client.data_sets.get_list_of_records(data_set_id=payload_data_set_id, limit=1, offset=0).result\n",
    "scoring_ids = [pl_records_resp[\"records\"][0][\"entity\"][\"values\"][\"scoring_id\"]]\n",
    "print(\"Running explanations on scoring IDs: {}\".format(scoring_ids))\n",
    "explanation_types = [\"lime\", \"contrastive\"]\n",
    "result = wos_client.monitor_instances.explanation_tasks(scoring_ids=scoring_ids, explanation_types=explanation_types, subscription_id=subscription_id).result\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom monitors and metrics <a name=\"custom\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register custom monitor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_definition(monitor_name):\n",
    "    monitor_definitions = wos_client.monitor_definitions.list().result.monitor_definitions\n",
    "    \n",
    "    for definition in monitor_definitions:\n",
    "        if monitor_name == definition.entity.name:\n",
    "            return definition\n",
    "    \n",
    "    return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================\n",
      "\n",
      " Waiting for end of adding monitor definition my_model_performance \n",
      "\n",
      "===================================================================\n",
      "\n",
      "\n",
      "\n",
      "finished\n",
      "\n",
      "-------------------------------------------------\n",
      " Successfully finished adding monitor definition \n",
      "-------------------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "monitor_name = 'my model performance'\n",
    "metrics = [MonitorMetricRequest(name='sensitivity',\n",
    "                                thresholds=[MetricThreshold(type=MetricThresholdTypes.LOWER_LIMIT, default=0.8)]),\n",
    "          MonitorMetricRequest(name='specificity',\n",
    "                                thresholds=[MetricThreshold(type=MetricThresholdTypes.LOWER_LIMIT, default=0.75)])]\n",
    "tags = [MonitorTagRequest(name='region', description='customer geographical region')]\n",
    "\n",
    "existing_definition = get_definition(monitor_name)\n",
    "\n",
    "if existing_definition is None:\n",
    "    custom_monitor_details = wos_client.monitor_definitions.add(name=monitor_name, metrics=metrics, tags=tags, background_mode=False).result\n",
    "else:\n",
    "    custom_monitor_details = existing_definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show available monitors types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Monitor definitions</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>monitor id</th><th style='border: 1px solid #dddddd'>monitor name</th><th style='border: 1px solid #dddddd'>metrics names</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>my_model_performance</td><td style='border: 1px solid #dddddd'>my model performance</td><td style='border: 1px solid #dddddd'>['sensitivity', 'specificity']</td></tr><tr><td style='border: 1px solid #dddddd'>fairness</td><td style='border: 1px solid #dddddd'>Fairness</td><td style='border: 1px solid #dddddd'>['Fairness value', 'Average odds difference', 'False discovery rate difference', 'Error rate difference', 'False negative rate difference', 'False omission rate difference', 'False positive rate difference', 'True positive rate difference', 'Average absolute odds difference', 'Statistical parity difference', 'Impact score']</td></tr><tr><td style='border: 1px solid #dddddd'>model_health</td><td style='border: 1px solid #dddddd'>Model health</td><td style='border: 1px solid #dddddd'>['Total scoring requests', 'Total records', 'Average records', 'Median records', 'Maximum records', 'Minimum records', 'Total payload size', 'Average payload size', 'Median payload size', 'Minimum payload size', 'Maximum payload size', 'Average API throughput', 'Minimum API throughput', 'Maximum API throughput', 'Median API throughput', 'Average API latency', 'Minimum API latency', 'Maximum API latency', 'Median API latency', 'Average record throughput', 'Minimum record throughput', 'Maximum record throughput', 'Median record throughput', 'Average record latency', 'Minimum record latency', 'Maximum record latency', 'Median record latency', 'Users', 'Errors', 'Data errors', 'System errors', 'Total input token count', 'Average input token count', 'Median input token count', 'Maximum input token count', 'Minimum input token count', 'Total output token count', 'Average output token count', 'Median output token count', 'Maximum output token count', 'Minimum output token count']</td></tr><tr><td style='border: 1px solid #dddddd'>data_health</td><td style='border: 1px solid #dddddd'>Data Health</td><td style='border: 1px solid #dddddd'>['Absence Count', 'Empty Strings', 'Data Type Mismatch', 'Class Confusion', 'Duplicate Rows', 'Unique Columns']</td></tr><tr><td style='border: 1px solid #dddddd'>performance</td><td style='border: 1px solid #dddddd'>Performance</td><td style='border: 1px solid #dddddd'>['Number of records']</td></tr><tr><td style='border: 1px solid #dddddd'>explainability</td><td style='border: 1px solid #dddddd'>Explainability</td><td style='border: 1px solid #dddddd'>['Global explanation stability']</td></tr><tr><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>Model risk management monitoring</td><td style='border: 1px solid #dddddd'>['Tests run', 'Tests passed', 'Tests failed', 'Tests skipped', 'Fairness score', 'Quality score', 'Drift score']</td></tr><tr><td style='border: 1px solid #dddddd'>generative_ai_quality</td><td style='border: 1px solid #dddddd'>Generative AI Quality</td><td style='border: 1px solid #dddddd'>['Records processed', 'ROUGE-1', 'ROUGE-2', 'ROUGE-L', 'ROUGE-Lsum', 'SARI', 'METEOR', 'F1 Score', 'Precision', 'Recall', 'BLEU', 'Jaccard similarity', 'Cosine similarity', 'Output data PII', 'Input data PII', 'Output data HAP', 'Input data HAP', 'Readability', 'Micro F1 Score', 'Macro F1 Score', 'Micro precision', 'Macro precision', 'Micro recall', 'Macro recall', 'Exact match', 'Faithfulness', 'Context relevance', 'Answer relevance', 'Unsuccessful requests']</td></tr><tr><td style='border: 1px solid #dddddd'>drift_v2</td><td style='border: 1px solid #dddddd'>Drift v2</td><td style='border: 1px solid #dddddd'>['Output drift', 'Prediction drift', 'Model quality drift', 'Feature drift', 'Input metadata drift', 'Output metadata drift', 'Records processed', 'New records processed']</td></tr><tr><td style='border: 1px solid #dddddd'>drift</td><td style='border: 1px solid #dddddd'>Drift</td><td style='border: 1px solid #dddddd'>['Drop in accuracy', 'Predicted accuracy', 'Drop in data consistency']</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: First 10 records were displayed.\n"
     ]
    }
   ],
   "source": [
    "wos_client.monitor_definitions.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get monitors uids and details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "my_model_performance\n"
     ]
    }
   ],
   "source": [
    "custom_monitor_id = custom_monitor_details.metadata.id\n",
    "\n",
    "print(custom_monitor_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Monitor definition details: {\n",
      "  \"metadata\": {\n",
      "    \"id\": \"my_model_performance\",\n",
      "    \"crn\": \"crn:v1:bluemix:public:aiopenscale:us-south:a/0a3c25959fab4ecea2768fa6b8d61595:80e6093f-5acf-4eb7-9da6-7ba9bf56a929:monitor_definition:my_model_performance\",\n",
      "    \"url\": \"/v2/monitor_definitions/my_model_performance\",\n",
      "    \"created_at\": \"2024-07-16T06:25:10.852000Z\",\n",
      "    \"created_by\": \"IBMid-662005298W\"\n",
      "  },\n",
      "  \"entity\": {\n",
      "    \"name\": \"my model performance\",\n",
      "    \"metrics\": [\n",
      "      {\n",
      "        \"name\": \"sensitivity\",\n",
      "        \"thresholds\": [\n",
      "          {\n",
      "            \"type\": \"lower_limit\",\n",
      "            \"default\": 0.8\n",
      "          }\n",
      "        ],\n",
      "        \"expected_direction\": \"increasing\",\n",
      "        \"id\": \"sensitivity\"\n",
      "      },\n",
      "      {\n",
      "        \"name\": \"specificity\",\n",
      "        \"thresholds\": [\n",
      "          {\n",
      "            \"type\": \"lower_limit\",\n",
      "            \"default\": 0.75\n",
      "          }\n",
      "        ],\n",
      "        \"expected_direction\": \"increasing\",\n",
      "        \"id\": \"specificity\"\n",
      "      }\n",
      "    ],\n",
      "    \"tags\": [\n",
      "      {\n",
      "        \"name\": \"region\",\n",
      "        \"description\": \"customer geographical region\",\n",
      "        \"id\": \"region\"\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "custom_monitor_details = wos_client.monitor_definitions.get(monitor_definition_id=custom_monitor_id).result\n",
    "print('Monitor definition details:', custom_monitor_details)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enable custom monitor for subscription"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation 75dd6aa9-1c17-431e-bc14-019feee5592d \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "target = Target(\n",
    "        target_type=TargetTypes.SUBSCRIPTION,\n",
    "        target_id=subscription_id\n",
    "    )\n",
    "\n",
    "thresholds = [MetricThresholdOverride(metric_id='sensitivity', type = MetricThresholdTypes.LOWER_LIMIT, value=0.9)]\n",
    "\n",
    "custom_monitor_instance_details = wos_client.monitor_instances.create(\n",
    "            data_mart_id=data_mart_id,\n",
    "            background_mode=False,\n",
    "            monitor_definition_id=custom_monitor_id,\n",
    "            target=target,\n",
    "            parameters={}\n",
    ").result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get monitor instance id and configuration details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_monitor_instance_id = custom_monitor_instance_details.metadata.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"metadata\": {\n",
      "    \"id\": \"75dd6aa9-1c17-431e-bc14-019feee5592d\",\n",
      "    \"crn\": \"crn:v1:bluemix:public:aiopenscale:us-south:a/0a3c25959fab4ecea2768fa6b8d61595:80e6093f-5acf-4eb7-9da6-7ba9bf56a929:monitor_instance:75dd6aa9-1c17-431e-bc14-019feee5592d\",\n",
      "    \"url\": \"/v2/monitor_instances/75dd6aa9-1c17-431e-bc14-019feee5592d\",\n",
      "    \"created_at\": \"2024-07-16T06:25:46.655000Z\",\n",
      "    \"created_by\": \"IBMid-662005298W\",\n",
      "    \"modified_at\": \"2024-07-16T06:25:46.786000Z\",\n",
      "    \"modified_by\": \"IBMid-662005298W\"\n",
      "  },\n",
      "  \"entity\": {\n",
      "    \"data_mart_id\": \"80e6093f-5acf-4eb7-9da6-7ba9bf56a929\",\n",
      "    \"monitor_definition_id\": \"my_model_performance\",\n",
      "    \"target\": {\n",
      "      \"target_type\": \"subscription\",\n",
      "      \"target_id\": \"f1a960ef-b474-43f9-8621-d0b1f2a4ef68\"\n",
      "    },\n",
      "    \"parameters\": {},\n",
      "    \"status\": {\n",
      "      \"state\": \"active\"\n",
      "    }\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "custom_monitor_instance_details = wos_client.monitor_instances.get(custom_monitor_instance_id).result\n",
    "print(custom_monitor_instance_details)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing custom metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"timestamp\": \"2024-07-16T06:25:53.155818Z\",\n",
      "  \"metrics\": [\n",
      "    {\n",
      "      \"specificity\": 0.78,\n",
      "      \"sensitivity\": 0.67,\n",
      "      \"region\": \"us-south\"\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime, timezone, timedelta\n",
    "from ibm_watson_openscale.base_classes.watson_open_scale_v2 import MonitorMeasurementRequest\n",
    "measurement_request = [MonitorMeasurementRequest(timestamp=datetime.now(timezone.utc), \n",
    "                                                 metrics=[{\"specificity\": 0.78, \"sensitivity\": 0.67, \"region\": \"us-south\"}])]\n",
    "print(measurement_request[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'measurement_id': '43acd6d2-8847-4cc7-bc0d-59c22d0932aa', 'metrics': [{'region': 'us-south', 'sensitivity': 0.67, 'specificity': 0.78}], 'timestamp': '2024-07-16T06:25:53.155818Z'}]\n"
     ]
    }
   ],
   "source": [
    "published_measurement_response = wos_client.monitor_instances.measurements.add(\n",
    "    monitor_instance_id=custom_monitor_instance_id,\n",
    "    monitor_measurement_request=measurement_request).result\n",
    "published_measurement_id = published_measurement_response[0][\"measurement_id\"]\n",
    "print(published_measurement_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List and get custom metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"metadata\": {\n",
      "    \"id\": \"43acd6d2-8847-4cc7-bc0d-59c22d0932aa\",\n",
      "    \"crn\": \"crn:v1:bluemix:public:aiopenscale:us-south:a/0a3c25959fab4ecea2768fa6b8d61595:80e6093f-5acf-4eb7-9da6-7ba9bf56a929:measurement:43acd6d2-8847-4cc7-bc0d-59c22d0932aa\",\n",
      "    \"url\": \"/v2/monitor_instances/75dd6aa9-1c17-431e-bc14-019feee5592d/measurements/43acd6d2-8847-4cc7-bc0d-59c22d0932aa\",\n",
      "    \"created_at\": \"2024-07-16T06:25:56.829050Z\",\n",
      "    \"created_by\": \"IBMid-662005298W\"\n",
      "  },\n",
      "  \"entity\": {\n",
      "    \"timestamp\": \"2024-07-16T06:25:53.155818Z\",\n",
      "    \"values\": [\n",
      "      {\n",
      "        \"metrics\": [\n",
      "          {\n",
      "            \"id\": \"sensitivity\",\n",
      "            \"value\": 0.67\n",
      "          },\n",
      "          {\n",
      "            \"id\": \"specificity\",\n",
      "            \"value\": 0.78\n",
      "          }\n",
      "        ],\n",
      "        \"tags\": [\n",
      "          {\n",
      "            \"id\": \"region\",\n",
      "            \"value\": \"us-south\"\n",
      "          }\n",
      "        ]\n",
      "      }\n",
      "    ],\n",
      "    \"issue_count\": 0,\n",
      "    \"target\": {\n",
      "      \"target_type\": \"subscription\",\n",
      "      \"target_id\": \"f1a960ef-b474-43f9-8621-d0b1f2a4ef68\"\n",
      "    },\n",
      "    \"monitor_instance_id\": \"75dd6aa9-1c17-431e-bc14-019feee5592d\",\n",
      "    \"monitor_definition_id\": \"my_model_performance\"\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "time.sleep(5)\n",
    "published_measurement = wos_client.monitor_instances.measurements.get(monitor_instance_id=custom_monitor_instance_id, measurement_id=published_measurement_id).result\n",
    "print(published_measurement)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Historical data <a name=\"historical\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "historyDays = 7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical fairness metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm: history_fairness_v2.json: No such file or directory\n",
      "-rw-r--r--  1 nelwin  staff   1.1M Jul 16 11:56 history_fairness_v2.json\n"
     ]
    }
   ],
   "source": [
    "!rm history_fairness_v2.json\n",
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/historical_data/credit_risk/history_fairness_v2.json\n",
    "!ls -lh history_fairness_v2.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading day 1\n",
      "Loading day 2\n",
      "Loading day 3\n",
      "Loading day 4\n",
      "Loading day 5\n",
      "Loading day 6\n",
      "Loading day 7\n",
      "Finished\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime, timedelta, timezone\n",
    "\n",
    "from ibm_watson_openscale.base_classes.watson_open_scale_v2 import Source\n",
    "from ibm_watson_openscale.base_classes.watson_open_scale_v2 import Measurements\n",
    "with open(\"history_fairness_v2.json\") as f:\n",
    "    fairness_values = json.load(f)\n",
    "    for day in range(historyDays):\n",
    "        print('Loading day', day + 1)\n",
    "        daily_measurement_requests = []\n",
    "        sources_list = []\n",
    "        for hour in range(24):\n",
    "            score_time = (datetime.now(timezone.utc) + timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n",
    "            index = (day * 24 + hour) % len(fairness_values) # wrap around and reuse values if needed\n",
    "            fairness_values[index][\"timestamp\"] = score_time\n",
    "            #print(score_time) \n",
    "            fairness_value = fairness_values[index]\n",
    "            metrics_list = fairness_value[\"metrics\"]\n",
    "            sources = fairness_value[\"sources\"]\n",
    "            sources_list = []\n",
    "            for source in sources:\n",
    "                source_id = source[\"id\"]\n",
    "                source_type = source[\"type\"]\n",
    "                source_data = source[\"data\"]\n",
    "                if source_id == \"bias_detection_summary\":\n",
    "                    source_data[\"evaluated_at\"] = score_time\n",
    "                    source_data[\"favourable_class\"] = [\"No Risk\"]\n",
    "                    source_data[\"unfavourable_class\"] = [\"Risk\"]\n",
    "                    source_data[\"score_type\"] = \"disparate impact\"\n",
    "                sources_list.append(\n",
    "                    Source(\n",
    "                        id=source_id,\n",
    "                        type=source_type,\n",
    "                        data=source_data\n",
    "                    )\n",
    "                )  \n",
    "            measurement_request = MonitorMeasurementRequest(metrics=metrics_list, sources=sources_list, timestamp=score_time)\n",
    "            daily_measurement_requests.append(measurement_request)\n",
    "        measurements_client = Measurements(wos_client)\n",
    "        measurements_client.add(monitor_instance_id=fairness_monitor_instance_id, monitor_measurement_request=daily_measurement_requests)     \n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical debias metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm: history_debias_v2.json: No such file or directory\n",
      "-rw-r--r--  1 nelwin  staff   725K Jul 16 11:56 history_debias_v2.json\n"
     ]
    }
   ],
   "source": [
    "!rm history_debias_v2.json\n",
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/historical_data/credit_risk/history_debias_v2.json\n",
    "!ls -lh history_debias_v2.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading day 1\n",
      "Loading day 2\n",
      "Loading day 3\n",
      "Loading day 4\n",
      "Loading day 5\n",
      "Loading day 6\n",
      "Loading day 7\n",
      "Finished\n"
     ]
    }
   ],
   "source": [
    "with open(\"history_debias_v2.json\") as f:\n",
    "    debias_values = json.load(f)\n",
    "    for day in range(historyDays):\n",
    "        print('Loading day', day + 1)\n",
    "        daily_measurement_requests = []\n",
    "        sources_list = []\n",
    "        for hour in range(24):\n",
    "            score_time = (datetime.now(timezone.utc) + timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')\n",
    "            index = (day * 24 + hour) % len(debias_values) # wrap around and reuse values if needed\n",
    "            debias_values[index][\"timestamp\"] = score_time\n",
    "            debias_value = debias_values[index]\n",
    "            metrics_list = debias_value[\"metrics\"]\n",
    "            sources = debias_value[\"sources\"]\n",
    "            sources_list = []\n",
    "            for source in sources:\n",
    "                sources_list.append(\n",
    "                    Source(\n",
    "                        id=source[\"id\"],\n",
    "                        type=source[\"type\"],\n",
    "                        data=source[\"data\"]\n",
    "                    )\n",
    "                )  \n",
    "            measurement_request = MonitorMeasurementRequest(metrics=metrics_list, sources=sources_list, timestamp=score_time)\n",
    "            daily_measurement_requests.append(measurement_request)\n",
    "        measurements_client = Measurements(wos_client)\n",
    "        measurements_client.add(monitor_instance_id=fairness_monitor_instance_id, monitor_measurement_request=daily_measurement_requests)         \n",
    "\n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical quality metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading day 1\n",
      "Loading day 2\n",
      "Loading day 3\n",
      "Loading day 4\n",
      "Loading day 5\n",
      "Loading day 6\n",
      "Loading day 7\n",
      "Finished\n"
     ]
    }
   ],
   "source": [
    "measurements = [0.76, 0.78, 0.68, 0.72, 0.73, 0.77, 0.80]\n",
    "for day in range(historyDays):\n",
    "    quality_measurement_requests = []\n",
    "    print('Loading day', day + 1)\n",
    "    for hour in range(24):\n",
    "        score_time = datetime.utcnow() + timedelta(hours=(-(24*day + hour + 1)))\n",
    "        score_time = score_time.isoformat() + \"Z\"\n",
    "        \n",
    "        metric = {\"area_under_roc\": measurements[day]}\n",
    "                \n",
    "        measurement_request = MonitorMeasurementRequest(timestamp=score_time,metrics = [metric])\n",
    "        quality_measurement_requests.append(measurement_request)\n",
    "        \n",
    "        \n",
    "    response = wos_client.monitor_instances.measurements.add(\n",
    "                                            monitor_instance_id=quality_monitor_instance_id,\n",
    "                                            monitor_measurement_request=quality_measurement_requests).result    \n",
    "    \n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical confusion matrixes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm: history_quality_metrics.json: No such file or directory\n",
      "-rw-r--r--  1 nelwin  staff    78K Jul 16 11:56 history_quality_metrics.json\n"
     ]
    }
   ],
   "source": [
    "!rm history_quality_metrics.json\n",
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/historical_data/credit_risk/history_quality_metrics.json\n",
    "!ls -lh history_quality_metrics.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading day 1\n",
      "Loading day 2\n",
      "Loading day 3\n",
      "Loading day 4\n",
      "Loading day 5\n",
      "Loading day 6\n",
      "Loading day 7\n",
      "Finished\n"
     ]
    }
   ],
   "source": [
    "from ibm_watson_openscale.base_classes.watson_open_scale_v2 import Source\n",
    "\n",
    "with open('history_quality_metrics.json') as json_file:\n",
    "    records = json.load(json_file)\n",
    "    \n",
    "for day in range(historyDays):\n",
    "    index = 0\n",
    "    cm_measurement_requests = []\n",
    "    print('Loading day', day + 1)\n",
    "    \n",
    "    for hour in range(24):\n",
    "        score_time = datetime.utcnow() + timedelta(hours=(-(24*day + hour + 1)))\n",
    "        score_time = score_time.isoformat() + \"Z\"\n",
    "\n",
    "        metric = records[index]['metrics']\n",
    "        source = records[index]['sources']\n",
    "\n",
    "        \n",
    "        measurement_request = {\"timestamp\": score_time, \"metrics\": [metric], \"sources\": [source]}\n",
    "        cm_measurement_requests.append(measurement_request)\n",
    "\n",
    "        index+=1\n",
    "\n",
    "    response = wos_client.monitor_instances.measurements.add(monitor_instance_id=quality_monitor_instance_id, monitor_measurement_request=cm_measurement_requests).result    \n",
    "\n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical performance metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation 924ec8e9-266a-4b46-8f61-5975e1929b2a \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "target = Target(\n",
    "        target_type=TargetTypes.INSTANCE,\n",
    "        target_id=payload_data_set_id\n",
    "    )\n",
    "\n",
    "\n",
    "performance_monitor_instance_details = wos_client.monitor_instances.create(\n",
    "            data_mart_id=data_mart_id,\n",
    "            background_mode=False,\n",
    "            monitor_definition_id=wos_client.monitor_definitions.MONITORS.PERFORMANCE.ID,\n",
    "            target=target,\n",
    "            parameters={}\n",
    ").result\n",
    "performance_monitor_instance_id = performance_monitor_instance_details.metadata.id\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading day 1\n",
      "Loading day 2\n",
      "Loading day 3\n",
      "Loading day 4\n",
      "Loading day 5\n",
      "Loading day 6\n",
      "Loading day 7\n",
      "Finished\n"
     ]
    }
   ],
   "source": [
    "for day in range(historyDays):\n",
    "    performance_measurement_requests = []\n",
    "    print('Loading day', day + 1)\n",
    "    for hour in range(24):\n",
    "        score_time = datetime.utcnow() + timedelta(hours=(-(24*day + hour + 1)))\n",
    "        score_time = score_time.isoformat() + \"Z\"\n",
    "        score_count = random.randint(60, 600)\n",
    "        \n",
    "        metric = {\"record_count\": score_count, \"data_set_type\": \"scoring_payload\"}\n",
    "        \n",
    "        measurement_request = {\"timestamp\": score_time, \"metrics\": [metric]}\n",
    "        performance_measurement_requests.append(measurement_request)\n",
    "        \n",
    "    response = wos_client.monitor_instances.measurements.add(\n",
    "                                            monitor_instance_id=performance_monitor_instance_id,\n",
    "                                            monitor_measurement_request=performance_measurement_requests).result    \n",
    "\n",
    "print('Finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert historical drift measurements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--  1 nelwin  staff   831K Jul 16 11:56 history_drift_measurement_0.json\n",
      "-rw-r--r--  1 nelwin  staff   867K Jul 16 11:56 history_drift_measurement_1.json\n",
      "-rw-r--r--  1 nelwin  staff   870K Jul 16 11:56 history_drift_measurement_2.json\n",
      "-rw-r--r--  1 nelwin  staff   909K Jul 16 11:56 history_drift_measurement_3.json\n",
      "-rw-r--r--  1 nelwin  staff   840K Jul 16 11:56 history_drift_measurement_4.json\n",
      "-rw-r--r--  1 nelwin  staff   835K Jul 16 11:56 history_drift_measurement_5.json\n",
      "-rw-r--r--  1 nelwin  staff   840K Jul 16 11:56 history_drift_measurement_6.json\n"
     ]
    }
   ],
   "source": [
    "with io.capture_output() as captured:\n",
    "    !wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/historical_data/credit_risk/history_drift_measurement_0.json\n",
    "    !wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/historical_data/credit_risk/history_drift_measurement_1.json\n",
    "    !wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/historical_data/credit_risk/history_drift_measurement_2.json\n",
    "    !wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/historical_data/credit_risk/history_drift_measurement_3.json\n",
    "    !wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/historical_data/credit_risk/history_drift_measurement_4.json\n",
    "    !wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/historical_data/credit_risk/history_drift_measurement_5.json\n",
    "    !wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/historical_data/credit_risk/history_drift_measurement_6.json\n",
    "!ls -lh history_drift_measurement_*.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading day 1\n",
      "Daily loading finished.\n",
      "Loading day 2\n",
      "Daily loading finished.\n",
      "Loading day 3\n",
      "Daily loading finished.\n",
      "Loading day 4\n",
      "Daily loading finished.\n",
      "Loading day 5\n",
      "Daily loading finished.\n",
      "Loading day 6\n",
      "Daily loading finished.\n",
      "Loading day 7\n",
      "Daily loading finished.\n"
     ]
    }
   ],
   "source": [
    "for day in range(historyDays):\n",
    "    drift_measurements = []\n",
    "\n",
    "    with open(\"history_drift_measurement_{}.json\".format(day), 'r') as history_file:\n",
    "        drift_daily_measurements = json.load(history_file)\n",
    "    print('Loading day', day + 1)\n",
    "\n",
    "    #Historical data contains 8 records per day - each represents 3 hour drift window.\n",
    "    \n",
    "    for nb_window, records in enumerate(drift_daily_measurements):\n",
    "        for record in records:\n",
    "            window_start =  datetime.utcnow() + timedelta(hours=(-(24 * day + (nb_window+1)*3 + 1))) # first_payload_record_timestamp_in_window (oldest)\n",
    "            window_end = datetime.utcnow() + timedelta(hours=(-(24 * day + nb_window*3 + 1)))# last_payload_record_timestamp_in_window (most recent)\n",
    "            #modify start and end time for each record\n",
    "            record['sources'][0]['data']['start'] = window_start.isoformat() + \"Z\"\n",
    "            record['sources'][0]['data']['end'] = window_end.isoformat() + \"Z\"\n",
    "            \n",
    "            \n",
    "            metric = record['metrics'][0]\n",
    "            source = record['sources'][0]\n",
    "\n",
    "            measurement_request = {\"timestamp\": window_start.isoformat() + \"Z\", \"metrics\": [metric], \"sources\": [source]}\n",
    "            \n",
    "            drift_measurements.append(measurement_request)\n",
    "        \n",
    "    response = wos_client.monitor_instances.measurements.add(\n",
    "                                            monitor_instance_id=drift_monitor_instance_id,\n",
    "                                            monitor_measurement_request=drift_measurements).result    \n",
    "\n",
    "    \n",
    "    print(\"Daily loading finished.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional data to help debugging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Datamart: 80e6093f-5acf-4eb7-9da6-7ba9bf56a929\n",
      "Model: ce0b71a4-ecee-4e0d-b51a-123e0fad4acd\n",
      "Deployment: db664cd4-2467-4975-b314-e24115c37ea5\n"
     ]
    }
   ],
   "source": [
    "print('Datamart:', data_mart_id)\n",
    "print('Model:', model_uid)\n",
    "print('Deployment:', deployment_uid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identify transactions for Explainability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transaction IDs identified by the cells below can be copied and pasted into the Explainability tab of the OpenScale dashboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Data Set 83433d8c-302f-4b6e-a1cb-0dc4b0745e90 Records</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>asset_revision</th><th style='border: 1px solid #dddddd'>LoanAmount</th><th style='border: 1px solid #dddddd'>scoring_id</th><th style='border: 1px solid #dddddd'>Sex_IX</th><th style='border: 1px solid #dddddd'>LoanPurpose_IX</th><th style='border: 1px solid #dddddd'>CheckingStatus_IX</th><th style='border: 1px solid #dddddd'>InstallmentPercent</th><th style='border: 1px solid #dddddd'>ExistingSavings</th><th style='border: 1px solid #dddddd'>scoring_timestamp</th><th style='border: 1px solid #dddddd'>prediction_probability</th><th style='border: 1px solid #dddddd'>features</th><th style='border: 1px solid #dddddd'>CheckingStatus</th><th style='border: 1px solid #dddddd'>LoanPurpose</th><th style='border: 1px solid #dddddd'>predictedLabel</th><th style='border: 1px solid #dddddd'>OwnsProperty_IX</th><th style='border: 1px solid #dddddd'>prediction</th><th style='border: 1px solid #dddddd'>Job</th><th style='border: 1px solid #dddddd'>InstallmentPlans</th><th style='border: 1px solid #dddddd'>ExistingSavings_IX</th><th style='border: 1px solid #dddddd'>CreditHistory_IX</th><th style='border: 1px solid #dddddd'>OthersOnLoan</th><th style='border: 1px solid #dddddd'>LoanDuration</th><th style='border: 1px solid #dddddd'>Age</th><th style='border: 1px solid #dddddd'>CreditHistory</th><th style='border: 1px solid #dddddd'>InstallmentPlans_IX</th><th style='border: 1px solid #dddddd'>CurrentResidenceDuration</th><th style='border: 1px solid #dddddd'>Telephone_IX</th><th style='border: 1px solid #dddddd'>EmploymentDuration_IX</th><th style='border: 1px solid #dddddd'>Job_IX</th><th style='border: 1px solid #dddddd'>ForeignWorker_IX</th><th style='border: 1px solid #dddddd'>ForeignWorker</th><th style='border: 1px solid #dddddd'>Housing</th><th style='border: 1px solid #dddddd'>Housing_IX</th><th style='border: 1px solid #dddddd'>ExistingCreditsCount</th><th style='border: 1px solid #dddddd'>deployment_id</th><th style='border: 1px solid #dddddd'>rawPrediction</th><th style='border: 1px solid #dddddd'>Telephone</th><th style='border: 1px solid #dddddd'>OthersOnLoan_IX</th><th style='border: 1px solid #dddddd'>Sex</th><th style='border: 1px solid #dddddd'>Dependents</th><th style='border: 1px solid #dddddd'>OwnsProperty</th><th style='border: 1px solid #dddddd'>EmploymentDuration</th><th style='border: 1px solid #dddddd'>probability</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>630</td><td style='border: 1px solid #dddddd'>7512f474-6c88-44bd-a591-24350774b1ad-1</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>6.0</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>less_100</td><td style='border: 1px solid #dddddd'>2024-07-16T06:18:47.201Z</td><td style='border: 1px solid #dddddd'>0.8866628721250828</td><td style='border: 1px solid #dddddd'>[1.0, 3.0, 6.0, 0.0, 3.0, 1.0, 0.0, 0.0, 2.0, 1.0, 1.0, 0.0, 0.0, 10.0, 630.0, 2.0, 2.0, 33.0, 1.0, 1.0]</td><td style='border: 1px solid #dddddd'>less_0</td><td style='border: 1px solid #dddddd'>vacation</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>unskilled</td><td style='border: 1px solid #dddddd'>bank</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>3.0</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>10</td><td style='border: 1px solid #dddddd'>33</td><td style='border: 1px solid #dddddd'>all_credits_paid_back</td><td style='border: 1px solid #dddddd'>2.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>3.0</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>rent</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>5b644de6-5b83-4930-ac85-c600b2c0ac4f</td><td style='border: 1px solid #dddddd'>[17.733257442501653, 2.2667425574983446]</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>female</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>less_1</td><td style='border: 1px solid #dddddd'>[0.8866628721250828, 0.11333712787491725]</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>1845</td><td style='border: 1px solid #dddddd'>7512f474-6c88-44bd-a591-24350774b1ad-10</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>9.0</td><td style='border: 1px solid #dddddd'>3.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>2024-07-16T06:18:47.201Z</td><td style='border: 1px solid #dddddd'>0.8184510088196267</td><td style='border: 1px solid #dddddd'>[3.0, 2.0, 9.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 22.0, 1845.0, 2.0, 2.0, 31.0, 1.0, 1.0]</td><td style='border: 1px solid #dddddd'>greater_200</td><td style='border: 1px solid #dddddd'>business</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>2.0</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>22</td><td style='border: 1px solid #dddddd'>31</td><td style='border: 1px solid #dddddd'>outstanding_credit</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>no</td><td style='border: 1px solid #dddddd'>rent</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>5b644de6-5b83-4930-ac85-c600b2c0ac4f</td><td style='border: 1px solid #dddddd'>[16.369020176392535, 3.6309798236074644]</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>4_to_7</td><td style='border: 1px solid #dddddd'>[0.8184510088196267, 0.1815489911803732]</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>3180</td><td style='border: 1px solid #dddddd'>7512f474-6c88-44bd-a591-24350774b1ad-100</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>500_to_1000</td><td style='border: 1px solid #dddddd'>2024-07-16T06:18:47.201Z</td><td style='border: 1px solid #dddddd'>0.707344075368278</td><td style='border: 1px solid #dddddd'>[0.0, 1.0, 0.0, 2.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 9.0, 3180.0, 2.0, 4.0, 31.0, 1.0, 1.0]</td><td style='border: 1px solid #dddddd'>no_checking</td><td style='border: 1px solid #dddddd'>car_new</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>unskilled</td><td style='border: 1px solid #dddddd'>stores</td><td style='border: 1px solid #dddddd'>2.0</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>9</td><td style='border: 1px solid #dddddd'>31</td><td style='border: 1px solid #dddddd'>credits_paid_to_date</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>5b644de6-5b83-4930-ac85-c600b2c0ac4f</td><td style='border: 1px solid #dddddd'>[14.146881507365562, 5.853118492634437]</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>car_other</td><td style='border: 1px solid #dddddd'>4_to_7</td><td style='border: 1px solid #dddddd'>[0.707344075368278, 0.29265592463172185]</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>311</td><td style='border: 1px solid #dddddd'>7512f474-6c88-44bd-a591-24350774b1ad-101</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>4.0</td><td style='border: 1px solid #dddddd'>3.0</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>500_to_1000</td><td style='border: 1px solid #dddddd'>2024-07-16T06:18:47.201Z</td><td style='border: 1px solid #dddddd'>0.7874175935565438</td><td style='border: 1px solid #dddddd'>[3.0, 1.0, 4.0, 2.0, 1.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 10.0, 311.0, 3.0, 2.0, 36.0, 1.0, 1.0]</td><td style='border: 1px solid #dddddd'>greater_200</td><td style='border: 1px solid #dddddd'>appliances</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>2.0</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>10</td><td style='border: 1px solid #dddddd'>36</td><td style='border: 1px solid #dddddd'>credits_paid_to_date</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>rent</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>5b644de6-5b83-4930-ac85-c600b2c0ac4f</td><td style='border: 1px solid #dddddd'>[15.748351871130874, 4.251648128869124]</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>female</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>car_other</td><td style='border: 1px solid #dddddd'>4_to_7</td><td style='border: 1px solid #dddddd'>[0.7874175935565438, 0.21258240644345622]</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>9300</td><td style='border: 1px solid #dddddd'>7512f474-6c88-44bd-a591-24350774b1ad-102</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>9.0</td><td style='border: 1px solid #dddddd'>2.0</td><td style='border: 1px solid #dddddd'>5</td><td style='border: 1px solid #dddddd'>500_to_1000</td><td style='border: 1px solid #dddddd'>2024-07-16T06:18:47.201Z</td><td style='border: 1px solid #dddddd'>0.578602331772885</td><td style='border: 1px solid #dddddd'>[2.0, 1.0, 9.0, 2.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 2.0, 1.0, 1.0, 25.0, 9300.0, 5.0, 3.0, 46.0, 3.0, 1.0]</td><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>business</td><td style='border: 1px solid #dddddd'>Risk</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>management_self-employed</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>2.0</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>co-applicant</td><td style='border: 1px solid #dddddd'>25</td><td style='border: 1px solid #dddddd'>46</td><td style='border: 1px solid #dddddd'>credits_paid_to_date</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>2.0</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>no</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>0.0</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>5b644de6-5b83-4930-ac85-c600b2c0ac4f</td><td style='border: 1px solid #dddddd'>[8.427953364542299, 11.5720466354577]</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>car_other</td><td style='border: 1px solid #dddddd'>4_to_7</td><td style='border: 1px solid #dddddd'>[0.42139766822711494, 0.578602331772885]</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wos_client.data_sets.show_records(payload_data_set_id, limit=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations!\n",
    "\n",
    "You have finished the hands-on lab for IBM Watson OpenScale. You can now view the [OpenScale Dashboard](https://aiopenscale.cloud.ibm.com/). Click on the tile for the German Credit model to see fairness, accuracy, and performance monitors. Click on the timeseries graph to get detailed information on transactions during a specific time window.\n",
    "\n",
    "## Next steps\n",
    "\n",
    "OpenScale shows model performance over time. You have two options to keep data flowing to your OpenScale graphs:\n",
    "  * Download, configure and schedule the [model feed notebook](https://raw.githubusercontent.com/emartensibm/german-credit/master/german_credit_scoring_feed.ipynb). This notebook can be set up with your WML credentials, and scheduled to provide a consistent flow of scoring requests to your model, which will appear in your OpenScale monitors.\n",
    "  * Re-run this notebook. Running this notebook from the beginning will delete and re-create the model and deployment, and re-create the historical data. Please note that the payload and measurement logs for the previous deployment will continue to be stored in your datamart, and can be deleted if necessary."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}