notebooks/AI OpenScale and SPSS C&DS Engine.ipynb

{
    "nbformat_minor": 2, 
    "cells": [
        {
            "source": "<img src=\"https://github.com/pmservice/ai-openscale-tutorials/raw/master/notebooks/images/banner.png\" align=\"left\" alt=\"banner\">", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "# Working with SPSS Collaboration and Deployment Services", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "This notebook shows how to log the payload for the model deployed on custom model serving engine using Watson OpenScale python sdk.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Contents\n - Setup\n - Binding machine learning engine\n - Subscriptions\n - Performance monitor, scoring and payload logging\n - Quality monitor and feedback logging\n - Fairness monitoring and explanations\n - Custom monitors and metrics", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## Setup", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Sample model creation using SPSS Modeler", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "- Download training data set from [here](https://github.com/pmservice/ai-openscale-tutorials/blob/master/notebooks/data/credit_risk_training.csv)\n- Download SPSS Modeler stream from [here](https://github.com/pmservice/ai-openscale-tutorials/blob/master/notebooks/data/german_credit_risk_tutorial.str)\n- Deploy the model using SPSS C&DS as web service", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Installation and authentication", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 1, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Successfully installed ibm-ai-openscale-2.1.7\r\n"
                }
            ], 
            "source": "!pip install --upgrade ibm-ai-openscale | tail -n 1"
        }, 
        {
            "source": "Import and initiate.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 2, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from ibm_ai_openscale import APIClient4ICP\nfrom ibm_ai_openscale.supporting_classes import PayloadRecord\nfrom ibm_ai_openscale.supporting_classes.enums import InputDataType, ProblemType\nfrom ibm_ai_openscale.engines import *\nfrom ibm_ai_openscale.utils import *\nimport urllib3\nimport os\n\nurllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)"
        }, 
        {
            "source": "#### Let's define some constants required to set up data mart:\n\n- AIOS_CREDENTIALS (ICP)\n- DATABASE_CREDENTIALS (DB2 on ICP)\n- SCHEMA_NAME", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 3, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "AIOS_CREDENTIALS = {\n    \"url\": \"***\",\n    \"username\": \"***\",\n    \"password\": \"***\"\n}"
        }, 
        {
            "execution_count": 4, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by Watson Studio for sharing."
        }, 
        {
            "execution_count": 5, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "DATABASE_CREDENTIALS = {\n    \"jdbcurl\": \"***\",\n    \"hostname\": \"***\",\n    \"username\": \"***\",\n    \"password\": \"***\",\n    \"port\": 50000,\n    \"db\": \"***\",\n    \"dsn\": \"***\",\n    \"uri\": \"***\"\n}"
        }, 
        {
            "execution_count": 6, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by Watson Studio for sharing."
        }, 
        {
            "execution_count": 7, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "{'db_type': 'db2', 'hostname': 'https://9.30.43.150', 'password': 'C0wTiger', 'port': 50000, 'db': 'SAMPLE', 'username': 'db2inst1', 'uri': 'db2://9.30.43.150:50000/SAMPLE:user=db2inst1;password=C0wTiger;'}\n"
                }
            ], 
            "source": "print(DATABASE_CREDENTIALS)"
        }, 
        {
            "execution_count": 26, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "SCHEMA_NAME = 'SPSSTF01'"
        }, 
        {
            "execution_count": 27, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "client = APIClient4ICP(AIOS_CREDENTIALS)"
        }, 
        {
            "execution_count": 28, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 28, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "'2.1.7'"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "client.version"
        }, 
        {
            "source": "### DataMart setup", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 29, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "#client.data_mart.delete()"
        }, 
        {
            "execution_count": 30, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "client.data_mart.setup(db_credentials=DATABASE_CREDENTIALS, schema=SCHEMA_NAME)"
        }, 
        {
            "execution_count": 31, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "{'database_configuration': {'database_type': 'db2', 'credentials': {'hostname': 'https://9.30.43.150', 'db_type': 'db2', 'username': 'db2inst1', 'uri': 'db2://9.30.43.150:50000/SAMPLE:user=db2inst1;password=C0wTiger;', 'db': 'SAMPLE', 'port': 50000, 'password': 'C0wTiger'}, 'location': {'schema': 'SPSSTF01'}, 'name': 'db2'}, 'status': {'state': 'active'}, 'service_instance_crn': 'N/A'}\n"
                }
            ], 
            "source": "data_mart_details = client.data_mart.get_details()\nprint(data_mart_details)"
        }, 
        {
            "source": "<a id=\"binding\"></a>\n## Bind machine learning engines", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Bind  `SPSS C&DS` machine learning engine\n\nProvide credentials using following fields:\n- `username`\n- `password`\n- `url`", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 32, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "SPSS_CDS_ENGINE_CREDENTIALS = {\n        \"username\": \"***\",\n        \"password\": \"***\",\n        \"url\": \"***\",\n    }"
        }, 
        {
            "execution_count": 33, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# The code was removed by Watson Studio for sharing."
        }, 
        {
            "execution_count": 34, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "BINDING_NAME = 'My SPSS C&DS engine'\nbinding_details = client.data_mart.bindings.get_details()\nbinding_uid = [binding[\"entity\"][\"instance_id\"] for binding in binding_details['service_bindings'] if binding[\"entity\"][\"name\"] == BINDING_NAME]\n\nif len(binding_uid) > 0:\n    [binding_uid] = binding_uid\nelse:\n    binding_uid = client.data_mart.bindings.add(BINDING_NAME, SPSSMachineLearningInstance(SPSS_CDS_ENGINE_CREDENTIALS))"
        }, 
        {
            "source": "<a id=\"subsciption\"></a>\n## Subscriptions", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Add subscriptions", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "List available deployments.\n\n**Note:** Depending on number of assets it may take some time.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 35, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<HTML>\n        <body>\n            <h3>Available assets (binding_uid='1d9297e2-5047-4b77-80a3-2fbca8f4fc87')</h3>\n            <table style='border: 1px solid #dddddd; font-family: Courier'>\n                <th style='border: 1px solid #dddddd'>source_uid</th><th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>created</th><th style='border: 1px solid #dddddd'>type</th><th style='border: 1px solid #dddddd'>frameworks</th><th style='border: 1px solid #dddddd'>binding_uid</th><th style='border: 1px solid #dddddd'>is_subscribed</th>\n                <tr><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8265</td><td style='border: 1px solid #dddddd'>ai_drug_mlp</td><td style='border: 1px solid #dddddd'>0000-01-01T00:00:00.0Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8272</td><td style='border: 1px solid #dddddd'>ai_drug_svm</td><td style='border: 1px solid #dddddd'>0000-01-01T00:00:00.0Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8258</td><td style='border: 1px solid #dddddd'>ai_drug_mlp_proba_BiasQA</td><td style='border: 1px solid #dddddd'>0000-01-01T00:00:00.0Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td><td style='border: 1px solid #dddddd'>0000-01-01T00:00:00.0Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8ad2</td><td style='border: 1px solid #dddddd'>diamonds</td><td style='border: 1px solid #dddddd'>0000-01-01T00:00:00.0Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd81f2</td><td style='border: 1px solid #dddddd'>NewTelcoChurnScore</td><td style='border: 1px solid #dddddd'>0000-01-01T00:00:00.0Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8ada</td><td style='border: 1px solid #dddddd'>employee</td><td style='border: 1px solid #dddddd'>0000-01-01T00:00:00.0Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8be2</td><td style='border: 1px solid #dddddd'>walking_activity</td><td style='border: 1px solid #dddddd'>0000-01-01T00:00:00.0Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8ae2</td><td style='border: 1px solid #dddddd'>ulcer-recurrence</td><td style='border: 1px solid #dddddd'>0000-01-01T00:00:00.0Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>False</td></tr><tr><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8ac9</td><td style='border: 1px solid #dddddd'>customer_visits</td><td style='border: 1px solid #dddddd'>0000-01-01T00:00:00.0Z</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>False</td></tr>\n            </table>\n        </body>\n        </HTML>", 
                        "text/plain": "<IPython.core.display.HTML object>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "client.data_mart.bindings.list_assets(binding_uid=binding_uid)"
        }, 
        {
            "source": "**Action:** Assign your credit risk model source_uid to `source_uid` variable below.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 36, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "MODEL_NAME='german_credit_risk_tutorial_BiasQA'\nasset_details = client.data_mart.bindings.get_asset_details(binding_uid=binding_uid)\nsource_uid = [asset[\"source_uid\"] for asset in asset_details if asset[\"name\"]==MODEL_NAME]\n\nif len(source_uid)>0:\n    [source_uid] = source_uid\nelse:\n    raise ValueError('Model with name \"{}\" not found.'.format(MODEL_NAME))"
        }, 
        {
            "execution_count": 37, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "subscription = client.data_mart.subscriptions.add(\n    SPSSMachineLearningAsset(source_uid=source_uid,\n                             binding_uid=binding_uid,\n                             input_data_type=InputDataType.STRUCTURED,\n                             problem_type=ProblemType.BINARY_CLASSIFICATION,\n                             label_column=\"Risk\",\n                             prediction_column=\"$N-Risk\",\n                             class_probability_columns=[\"$NP-No Risk\", \"$NP-Risk\"]))"
        }, 
        {
            "source": "#### Get subscriptions list", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 38, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "subscriptions = client.data_mart.subscriptions.get_details()"
        }, 
        {
            "execution_count": 39, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "['091e60170f73c4410000016a9a18aecd8242']\n"
                }
            ], 
            "source": "subscriptions_uids = client.data_mart.subscriptions.get_uids()\nprint(subscriptions_uids)"
        }, 
        {
            "source": "#### List subscriptions", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 40, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<HTML>\n        <body>\n            <h3>Subscriptions</h3>\n            <table style='border: 1px solid #dddddd; font-family: Courier'>\n                <th style='border: 1px solid #dddddd'>uid</th><th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>type</th><th style='border: 1px solid #dddddd'>binding_uid</th><th style='border: 1px solid #dddddd'>created</th>\n                <tr><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>2019-05-14T12:23:23.280Z</td></tr>\n            </table>\n        </body>\n        </HTML>", 
                        "text/plain": "<IPython.core.display.HTML object>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "client.data_mart.subscriptions.list()"
        }, 
        {
            "source": "## Performance monitor, scoring and payload logging", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Score the credit risk model and measure response time", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 41, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "{'providedBy': 'german_credit_risk_tutorial_BiasQA', 'id': '1575a603-43b5-470d-a61d-bf8cbe2fbac9', 'columnNames': {'name': ['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker', '$N-Risk', '$NC-Risk', '$NP-No Risk', '$NP-Risk']}, 'rowValues': [{'value': [{'value': '0_to_200'}, {'value': '31'}, {'value': 'credits_paid_to_date'}, {'value': 'other'}, {'value': '1889'}, {'value': '100_to_500'}, {'value': 'less_1'}, {'value': '3'}, {'value': 'female'}, {'value': 'none'}, {'value': '3'}, {'value': 'savings_insurance'}, {'value': '32'}, {'value': 'none'}, {'value': 'own'}, {'value': '1'}, {'value': 'skilled'}, {'value': '1'}, {'value': 'none'}, {'value': 'yes'}, {'value': 'No Risk'}, {'value': '0.8252855725848809'}, {'value': '0.8252855725848809'}, {'value': '0.17471442741511914'}]}]}\n"
                }
            ], 
            "source": "import requests\nfrom requests.auth import HTTPBasicAuth\nimport time\nimport json\n\nsubscription_details = subscription.get_details()\nscoring_endpoint = subscription_details['entity']['deployments'][0]['scoring_endpoint']['url']\ninput_table_id = subscription_details['entity']['asset_properties']['input_data_schema']['id']\nnode_id = subscription_details['entity']['asset']['name']\n\nscoring_payload = {'requestInputTable': [{'id': input_table_id, 'requestInputRow': [{'input': [\n            {'name': 'CheckingStatus', 'value': '0_to_200'}, {'name': 'LoanDuration', 'value': '31'},\n            {'name': 'CreditHistory', 'value': 'credits_paid_to_date'}, {'name': 'LoanPurpose', 'value': 'other'},\n            {'name': 'LoanAmount', 'value': '1889'}, {'name': 'ExistingSavings', 'value': '100_to_500'},\n            {'name': 'EmploymentDuration', 'value': 'less_1'}, {'name': 'InstallmentPercent', 'value': '3'},\n            {'name': 'Sex', 'value': 'female'}, {'name': 'OthersOnLoan', 'value': 'none'},\n            {'name': 'CurrentResidenceDuration', 'value': '3'}, {'name': 'OwnsProperty', 'value': 'savings_insurance'},\n            {'name': 'Age', 'value': '32'}, {'name': 'InstallmentPlans', 'value': 'none'},\n            {'name': 'Housing', 'value': 'own'}, {'name': 'ExistingCreditsCount', 'value': '1'},\n            {'name': 'Job', 'value': 'skilled'}, {'name': 'Dependents', 'value': '1'},\n            {'name': 'Telephone', 'value': 'none'}, {'name': 'ForeignWorker', 'value': 'yes'}]}]}], 'id': node_id}\n\nstart_time = time.time()\nresp_score = requests.post(url=scoring_endpoint, json=scoring_payload, auth=HTTPBasicAuth(username=SPSS_CDS_ENGINE_CREDENTIALS['username'], password=SPSS_CDS_ENGINE_CREDENTIALS['password']))\n\nresponse_time = int((time.time() - start_time)*1000)\nresult = resp_score.json()\n\nprint(result)"
        }, 
        {
            "source": "### Store the request and response in payload logging table", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Store the payload using Python SDK", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "**Hint:** You can embed payload logging code into your application so it is logged automatically each time you score the model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 42, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Waiting 10 seconds for propagation... payload data should be propagated.\n"
                }
            ], 
            "source": "records_list = [PayloadRecord(request=scoring_payload, response=result, response_time=response_time), \n                PayloadRecord(request=scoring_payload, response=result, response_time=response_time)]\n\nfor i in range(1, 10):\n    records_list.append(PayloadRecord(request=scoring_payload, response=result, response_time=response_time))\n\nsubscription.payload_logging.store(records=records_list)\n\nprint(\"Waiting 10 seconds for propagation...\", end=\" \")\ntime.sleep(10)\nprint('payload data should be propagated.')"
        }, 
        {
            "source": "### Performance metrics of scoring requests", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 43, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<HTML>\n        <body>\n            <h3>PerformanceMetrics (binding_id=1d9297e2-5047-4b77-80a3-2fbca8f4fc87, subscription_id=091e60170f73c4410000016a9a18aecd8242)</h3>\n            <table style='border: 1px solid #dddddd; font-family: Courier'>\n                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>scoring_time</th><th style='border: 1px solid #dddddd'>scoring_records</th><th style='border: 1px solid #dddddd'>binding_id</th><th style='border: 1px solid #dddddd'>subscription_id</th><th style='border: 1px solid #dddddd'>deployment_id</th><th style='border: 1px solid #dddddd'>process</th><th style='border: 1px solid #dddddd'>asset_revision</th>\n                <tr><td style='border: 1px solid #dddddd'>2019-05-14 12:24:16.846000+00:00</td><td style='border: 1px solid #dddddd'>23.0</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:24:16.846000+00:00</td><td style='border: 1px solid #dddddd'>23.0</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:24:16.846000+00:00</td><td style='border: 1px solid #dddddd'>23.0</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:24:16.846000+00:00</td><td style='border: 1px solid #dddddd'>23.0</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:24:16.845000+00:00</td><td style='border: 1px solid #dddddd'>23.0</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:24:16.845000+00:00</td><td style='border: 1px solid #dddddd'>23.0</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:24:16.845000+00:00</td><td style='border: 1px solid #dddddd'>23.0</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:24:16.845000+00:00</td><td style='border: 1px solid #dddddd'>23.0</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:24:16.845000+00:00</td><td style='border: 1px solid #dddddd'>23.0</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:24:16.845000+00:00</td><td style='border: 1px solid #dddddd'>23.0</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td></tr>\n            </table>\n        </body>\n        </HTML>", 
                        "text/plain": "<IPython.core.display.HTML object>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "subscription.performance_monitoring.show_table()"
        }, 
        {
            "source": "## Quality monitor and feedback logging", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Enable quality monitoring", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "You need to provide the monitoring `threshold` and `min_records` (minimal number of feedback records).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 44, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "subscription.quality_monitoring.enable(threshold=0.7, min_records=10)"
        }, 
        {
            "source": "### Feedback records logging", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Feedback records are used to evaluate your model. The predicted values are compared to real values (feedback records).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "You can check the schema of feedback table using below method.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 45, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<HTML>\n        <body>\n            <h3>Schema of Feedback_091e60170f73c4410000016a9a18aecd8242</h3>\n            <table style='border: 1px solid #dddddd; font-family: Courier'>\n                <th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>type</th><th style='border: 1px solid #dddddd'>nullable</th>\n                <tr><td style='border: 1px solid #dddddd'>CheckingStatus</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>LoanDuration</td><td style='border: 1px solid #dddddd'>integer</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>CreditHistory</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>LoanPurpose</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>LoanAmount</td><td style='border: 1px solid #dddddd'>integer</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>ExistingSavings</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>EmploymentDuration</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>InstallmentPercent</td><td style='border: 1px solid #dddddd'>integer</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>Sex</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>OthersOnLoan</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>CurrentResidenceDuration</td><td style='border: 1px solid #dddddd'>integer</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>OwnsProperty</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>Age</td><td style='border: 1px solid #dddddd'>integer</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>InstallmentPlans</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>Housing</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>ExistingCreditsCount</td><td style='border: 1px solid #dddddd'>integer</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>Job</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>Dependents</td><td style='border: 1px solid #dddddd'>integer</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>Telephone</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>ForeignWorker</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>Risk</td><td style='border: 1px solid #dddddd'>string</td><td style='border: 1px solid #dddddd'>True</td></tr><tr><td style='border: 1px solid #dddddd'>_training</td><td style='border: 1px solid #dddddd'>timestamp</td><td style='border: 1px solid #dddddd'>False</td></tr>\n            </table>\n        </body>\n        </HTML>", 
                        "text/plain": "<IPython.core.display.HTML object>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "subscription.feedback_logging.print_table_schema()"
        }, 
        {
            "source": "The feedback records can be send to feedback table using below code.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Store feedback using CSV format from string", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 46, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from ibm_ai_openscale.supporting_classes.enums import FeedbackFormat\n\nfeedback_records = \"0_to_200,31,credits_paid_to_date,other,1889,100_to_500,less_1,3,female,none,3,savings_insurance,32,none,own,1,skilled,1,none,yes,Risk\"\n\nfor i in range(0, 10):\n    feedback_records = feedback_records + '\\n' + feedback_records\n\nsubscription.feedback_logging.store(feedback_data=feedback_records, feedback_format=FeedbackFormat.CSV)"
        }, 
        {
            "source": "#### Store feedback using CSV format from file", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 47, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "--2019-05-14 12:26:48--  https://raw.githubusercontent.com/pmservice/wml-sample-models/master/spss/credit-risk/data/credit_risk_feedback.csv\nResolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.68.133\nConnecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.68.133|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 40768 (40K) [text/plain]\nSaving to: \u2018credit_risk_feedback.csv\u2019\n\ncredit_risk_feedbac 100%[===================>]  39.81K  --.-KB/s    in 0.05s   \n\n2019-05-14 12:26:48 (779 KB/s) - \u2018credit_risk_feedback.csv\u2019 saved [40768/40768]\n\n"
                }
            ], 
            "source": "!wget https://raw.githubusercontent.com/pmservice/wml-sample-models/master/spss/credit-risk/data/credit_risk_feedback.csv"
        }, 
        {
            "execution_count": 48, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "with open('credit_risk_feedback.csv', 'rb') as csvFile:\n    subscription.feedback_logging.store(feedback_data=csvFile, feedback_format=FeedbackFormat.CSV, data_header=True, data_delimiter=',')"
        }, 
        {
            "execution_count": 49, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "       LoanDuration    LoanAmount  InstallmentPercent  \\\ncount   1315.000000   1315.000000         1315.000000   \nmean      29.539163   2383.437262            3.058555   \nstd        5.494063   1391.763046            0.492324   \nmin        4.000000    250.000000            1.000000   \n25%       31.000000   1889.000000            3.000000   \n50%       31.000000   1889.000000            3.000000   \n75%       31.000000   1889.000000            3.000000   \nmax       61.000000  10430.000000            5.000000   \n\n       CurrentResidenceDuration          Age  ExistingCreditsCount  \\\ncount               1315.000000  1315.000000           1315.000000   \nmean                   3.034221    33.610646              1.128517   \nstd                    0.447775     5.316086              0.363143   \nmin                    1.000000    19.000000              1.000000   \n25%                    3.000000    32.000000              1.000000   \n50%                    3.000000    32.000000              1.000000   \n75%                    3.000000    32.000000              1.000000   \nmax                    5.000000    64.000000              3.000000   \n\n        Dependents  \ncount  1315.000000  \nmean      1.036502  \nstd       0.187607  \nmin       1.000000  \n25%       1.000000  \n50%       1.000000  \n75%       1.000000  \nmax       2.000000  \n"
                }
            ], 
            "source": "subscription.feedback_logging.describe_table()"
        }, 
        {
            "source": "### Run quality monitoring on demand", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "By default, quality monitoring is run on hourly schedule. You can also trigger it on demand using below code.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 50, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "\n\n================================================================================\n\n Waiting for end of quality monitoring run 48bba8d4-118e-45a1-bb52-ad2c094b0086 \n\n================================================================================\n\n\n\ninitializing.\nrunning\ncompleted\n\n---------------------------\n Successfully finished run \n---------------------------\n\n\n"
                }
            ], 
            "source": "run_details = subscription.quality_monitoring.run(background_mode=False)"
        }, 
        {
            "source": "### Show the quality metrics", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 51, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<HTML>\n        <body>\n            <h3>QualityMetrics (binding_id=1d9297e2-5047-4b77-80a3-2fbca8f4fc87, subscription_id=091e60170f73c4410000016a9a18aecd8242)</h3>\n            <table style='border: 1px solid #dddddd; font-family: Courier'>\n                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower limit</th><th style='border: 1px solid #dddddd'>upper limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>binding_id</th><th style='border: 1px solid #dddddd'>subscription_id</th><th style='border: 1px solid #dddddd'>deployment_id</th>\n                <tr><td style='border: 1px solid #dddddd'>2019-05-14 12:27:22.739000+00:00</td><td style='border: 1px solid #dddddd'>true_positive_rate</td><td style='border: 1px solid #dddddd'>214c5480-7d37-4470-bcb6-f00136b10526</td><td style='border: 1px solid #dddddd'>0.036720751494449186</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:27:22.739000+00:00</td><td style='border: 1px solid #dddddd'>area_under_roc</td><td style='border: 1px solid #dddddd'>214c5480-7d37-4470-bcb6-f00136b10526</td><td style='border: 1px solid #dddddd'>0.31002704241389123</td><td style='border: 1px solid #dddddd'>0.7</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:27:22.739000+00:00</td><td style='border: 1px solid #dddddd'>precision</td><td style='border: 1px solid #dddddd'>214c5480-7d37-4470-bcb6-f00136b10526</td><td style='border: 1px solid #dddddd'>0.4174757281553398</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:27:22.739000+00:00</td><td style='border: 1px solid #dddddd'>f1_measure</td><td style='border: 1px solid #dddddd'>214c5480-7d37-4470-bcb6-f00136b10526</td><td style='border: 1px solid #dddddd'>0.06750392464678179</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:27:22.739000+00:00</td><td style='border: 1px solid #dddddd'>accuracy</td><td style='border: 1px solid #dddddd'>214c5480-7d37-4470-bcb6-f00136b10526</td><td style='border: 1px solid #dddddd'>0.09657794676806084</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:27:22.739000+00:00</td><td style='border: 1px solid #dddddd'>log_loss</td><td style='border: 1px solid #dddddd'>214c5480-7d37-4470-bcb6-f00136b10526</td><td style='border: 1px solid #dddddd'>1.56717570703848</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:27:22.739000+00:00</td><td style='border: 1px solid #dddddd'>false_positive_rate</td><td style='border: 1px solid #dddddd'>214c5480-7d37-4470-bcb6-f00136b10526</td><td style='border: 1px solid #dddddd'>0.4166666666666667</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:27:22.739000+00:00</td><td style='border: 1px solid #dddddd'>area_under_pr</td><td style='border: 1px solid #dddddd'>214c5480-7d37-4470-bcb6-f00136b10526</td><td style='border: 1px solid #dddddd'>0.6453002137149899</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:27:22.739000+00:00</td><td style='border: 1px solid #dddddd'>recall</td><td style='border: 1px solid #dddddd'>214c5480-7d37-4470-bcb6-f00136b10526</td><td style='border: 1px solid #dddddd'>0.036720751494449186</td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'></td><td style='border: 1px solid #dddddd'>model_type: original</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td></tr>\n            </table>\n        </body>\n        </HTML>", 
                        "text/plain": "<IPython.core.display.HTML object>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "subscription.quality_monitoring.show_table()"
        }, 
        {
            "source": "Get all calculated metrics.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 52, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "deployment_uids = subscription.get_deployment_uids()"
        }, 
        {
            "execution_count": 53, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 53, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "[{'asset_id': '091e60170f73c4410000016a9a18aecd8242',\n  'binding_id': '1d9297e2-5047-4b77-80a3-2fbca8f4fc87',\n  'process': 'Accuracy_evaluation_48bba8d4-118e-45a1-bb52-ad2c094b0086',\n  'tags': [{'id': 'model_type', 'value': 'original'}],\n  'ts': '2019-05-14T12:27:22.739Z',\n  'measurement_id': '214c5480-7d37-4470-bcb6-f00136b10526',\n  'monitor_definition_id': 'quality',\n  'subscription_id': '091e60170f73c4410000016a9a18aecd8242',\n  'metrics': [{'id': 'true_positive_rate', 'value': 0.036720751494449186},\n   {'lower_limit': 0.7, 'id': 'area_under_roc', 'value': 0.31002704241389123},\n   {'id': 'precision', 'value': 0.4174757281553398},\n   {'id': 'f1_measure', 'value': 0.06750392464678179},\n   {'id': 'accuracy', 'value': 0.09657794676806084},\n   {'id': 'log_loss', 'value': 1.56717570703848},\n   {'id': 'false_positive_rate', 'value': 0.4166666666666667},\n   {'id': 'area_under_pr', 'value': 0.6453002137149899},\n   {'id': 'recall', 'value': 0.036720751494449186}]}]"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "subscription.quality_monitoring.get_metrics(deployment_uid=deployment_uids[0])"
        }, 
        {
            "source": "## Fairness monitoring and explanations", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Enable and run fairness monitoring", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 54, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "import pandas as pd\nimport numpy as np"
        }, 
        {
            "execution_count": 55, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "training_data_df = pd.read_csv(\"https://raw.githubusercontent.com/pmservice/wml-sample-models/master/spss/credit-risk/data/credit_risk_training.csv\",\n                               dtype={'CheckingStatus': str, 'LoanDuration': int, 'CreditHistory': str, 'LoanPurpose': str,\n                                      'LoanAmount': int, 'ExistingSavings': str, 'EmploymentDuration': str, 'InstallmentPercent': int,\n                                      'Sex': str, 'OthersOnLoan': str, 'CurrentResidenceDuration': int, 'OwnsProperty': str,\n                                      'Age': int, 'InstallmentPlans': str, 'Housing': str, 'ExistingCreditsCount': int,\n                                      'Job': str, 'Dependents': int, 'Telephone': str, 'ForeignWorker': str, 'Risk': str})"
        }, 
        {
            "execution_count": 56, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker']\n['CheckingStatus', 'CreditHistory', 'LoanPurpose', 'ExistingSavings', 'EmploymentDuration', 'Sex', 'OthersOnLoan', 'OwnsProperty', 'InstallmentPlans', 'Housing', 'Job', 'Telephone', 'ForeignWorker']\n"
                }
            ], 
            "source": "feature_columns = [col for col in list(training_data_df.columns) if col!=\"Risk\"]\nprint(feature_columns)\n\ncategorical_features = [col for col, col_type in training_data_df.dtypes.to_dict().items() if col_type is np.dtype('O') and col!=\"Risk\"]\nprint(categorical_features)"
        }, 
        {
            "execution_count": 57, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "subscription.update(feature_columns=feature_columns, categorical_columns=categorical_features)"
        }, 
        {
            "execution_count": 58, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from ibm_ai_openscale.supporting_classes import Feature\n\nsubscription.fairness_monitoring.enable(\n            features=[\n                Feature(\"Sex\", majority=['male'], minority=['female'], threshold=0.95),\n                Feature(\"Age\", majority=[[26, 75]], minority=[[18, 25]], threshold=0.95)\n            ],\n            favourable_classes=['No Risk'],\n            unfavourable_classes=['Risk'],\n            min_records=4,\n            training_data=training_data_df\n        )"
        }, 
        {
            "execution_count": 59, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "\n\n=====================================================================\n\n Counting bias for deployment_uid=german_credit_risk_tutorial_BiasQA \n\n=====================================================================\n\n\n\nRUNNING..\nFINISHED\n\n---------------------------\n Successfully finished run \n---------------------------\n\n\n"
                }
            ], 
            "source": "fairness_run = subscription.fairness_monitoring.run(background_mode=False)"
        }, 
        {
            "source": "### Check fairness run results", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 60, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/html": "<HTML>\n        <body>\n            <h3>FairnessMetrics (binding_id=1d9297e2-5047-4b77-80a3-2fbca8f4fc87, subscription_id=091e60170f73c4410000016a9a18aecd8242)</h3>\n            <table style='border: 1px solid #dddddd; font-family: Courier'>\n                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>feature</th><th style='border: 1px solid #dddddd'>feature_value</th><th style='border: 1px solid #dddddd'>fairness_biased</th><th style='border: 1px solid #dddddd'>fairness_value</th><th style='border: 1px solid #dddddd'>fairness_fav_class</th><th style='border: 1px solid #dddddd'>binding_id</th><th style='border: 1px solid #dddddd'>subscription_id</th><th style='border: 1px solid #dddddd'>asset_revision</th><th style='border: 1px solid #dddddd'>deployment_id</th><th style='border: 1px solid #dddddd'>process</th>\n                <tr><td style='border: 1px solid #dddddd'>2019-05-14 12:29:41.325992+00:00</td><td style='border: 1px solid #dddddd'>Sex</td><td style='border: 1px solid #dddddd'>female</td><td style='border: 1px solid #dddddd'>False</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>100.0</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td><td style='border: 1px solid #dddddd'></td></tr><tr><td style='border: 1px solid #dddddd'>2019-05-14 12:29:41.325992+00:00</td><td style='border: 1px solid #dddddd'>Age</td><td style='border: 1px solid #dddddd'>[18, 25]</td><td style='border: 1px solid #dddddd'>False</td><td style='border: 1px solid #dddddd'>1.0</td><td style='border: 1px solid #dddddd'>100.0</td><td style='border: 1px solid #dddddd'>1d9297e2-5047-4b77-80a3-2fbca8f4fc87</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>091e60170f73c4410000016a9a18aecd8242</td><td style='border: 1px solid #dddddd'>german_credit_risk_tutorial_BiasQA</td><td style='border: 1px solid #dddddd'></td></tr>\n            </table>\n        </body>\n        </HTML>", 
                        "text/plain": "<IPython.core.display.HTML object>"
                    }, 
                    "metadata": {}
                }
            ], 
            "source": "subscription.fairness_monitoring.show_table()"
        }, 
        {
            "execution_count": 61, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "{'fields': ['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker'], 'values': [['0_to_200', 31, 'credits_paid_to_date', 'other', 1889, '100_to_500', 'less_1', 3, 'female', 'none', 3, 'savings_insurance', 32, 'none', 'own', 1, 'skilled', 1, 'none', 'yes']]}\n"
                }
            ], 
            "source": "scoring_payload = {\"fields\": [v for v in training_data_df.columns.values.tolist() if v != 'Risk'], \n                   \"values\": training_data_df.iloc[0:1,0:-1].values.tolist()}\nprint(scoring_payload)"
        }, 
        {
            "source": "### Explainability configuration and run", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Enable explainability", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 63, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "subscription.explainability.enable(training_data=training_data_df)"
        }, 
        {
            "source": "#### Get sample transaction_id from payload logging table (`scoring_id`)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 64, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "3fe9b98e-259a-456a-9be1-b1765030ae26-1\n"
                }
            ], 
            "source": "transaction_id = subscription.payload_logging.get_table_content(limit=1)['scoring_id'].values[0]\nprint(transaction_id)"
        }, 
        {
            "source": "#### Run explanation for sample `transaction_id`", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 65, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "\n\n====================================================================\n\n Looking for explanation for 3fe9b98e-259a-456a-9be1-b1765030ae26-1 \n\n====================================================================\n\n\n\nin_progress...................\nfinished\n\n---------------------------\n Successfully finished run \n---------------------------\n\n\n"
                }
            ], 
            "source": "explain_run = subscription.explainability.run(transaction_id=transaction_id, background_mode=False)"
        }, 
        {
            "execution_count": 80, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from tabulate import tabulate\nfrom IPython.core.display import HTML"
        }, 
        {
            "execution_count": 85, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 85, 
                    "metadata": {}, 
                    "data": {
                        "text/html": "<table>\n<thead>\n<tr><th>feature             </th><th style=\"text-align: right;\">  importance</th></tr>\n</thead>\n<tbody>\n<tr><td>LoanDuration        </td><td style=\"text-align: right;\">       -0.19</td></tr>\n<tr><td>ForeignWorker       </td><td style=\"text-align: right;\">        0.15</td></tr>\n<tr><td>OthersOnLoan        </td><td style=\"text-align: right;\">        0.14</td></tr>\n<tr><td>EmploymentDuration  </td><td style=\"text-align: right;\">        0.12</td></tr>\n<tr><td>ExistingCreditsCount</td><td style=\"text-align: right;\">        0.11</td></tr>\n<tr><td>InstallmentPlans    </td><td style=\"text-align: right;\">       -0.08</td></tr>\n<tr><td>CreditHistory       </td><td style=\"text-align: right;\">       -0.06</td></tr>\n<tr><td>Housing             </td><td style=\"text-align: right;\">        0.06</td></tr>\n<tr><td>Job                 </td><td style=\"text-align: right;\">       -0.06</td></tr>\n<tr><td>Telephone           </td><td style=\"text-align: right;\">        0.04</td></tr>\n</tbody>\n</table>", 
                        "text/plain": "<IPython.core.display.HTML object>"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "explain_result = pd.DataFrame.from_dict(explain_run['entity']['predictions'][0]['explanation_features'])\nHTML(tabulate(explain_result[['feature_name', 'weight']], headers=['feature', 'importance'] , showindex=False, floatfmt=\".2f\", tablefmt='html'))"
        }, 
        {
            "source": "---", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## Custom monitors and metrics", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Register custom monitor", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "def get_definition(monitor_name):\n    monitors_definitions = ai_client.data_mart.monitors.get_details()['monitor_definitions']\n    \n    for definition in monitors_definitions:\n        if monitor_name == definition['entity']['name']:\n            return definition\n    \n    return None"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from ibm_ai_openscale.supporting_classes import Metric, Tag\n\nmonitor_name = 'my model performance'\nmetrics = [Metric(name='sensitivity', lower_limit_default=0.8), Metric(name='specificity', lower_limit_default=0.75)]\ntags = [Tag(name='region', description='customer geographical region')]\n\nexisting_definition = get_definition(monitor_name)\n\nif existing_definition is None:\n    my_monitor = ai_client.data_mart.monitors.add(name=monitor_name, metrics=metrics, tags=tags)\nelse:\n    my_monitor = existing_definition"
        }, 
        {
            "source": "### List monitors", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Get monitors uids and details", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "monitor_uid = my_monitor['metadata']['guid']\nprint(monitor_uid)"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "my_monitor = ai_client.data_mart.monitors.get_details(monitor_uid=monitor_uid)\nprint('monitor definition details', my_monitor)"
        }, 
        {
            "source": "### Enable custom monitor for subscription", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from ibm_ai_openscale.supporting_classes import Threshold\n\nthresholds = [Threshold(metric_uid='sensitivity', lower_limit=0.9)]\nsubscription.monitoring.enable(monitor_uid=monitor_uid, thresholds=thresholds)"
        }, 
        {
            "source": "### Get monitor configuration details", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "subscription.monitoring.get_details(monitor_uid=monitor_uid)"
        }, 
        {
            "source": "### Storing custom metrics", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "metrics = {\"specificity\": 0.78, \"sensitivity\": 0.67}\n\nsubscription.monitoring.store_metrics(monitor_uid=monitor_uid, metrics=metrics)"
        }, 
        {
            "source": "### List and get custom metrics", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "subscription.monitoring.show_table(monitor_uid=monitor_uid)"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "custom_metrics = subscription.monitoring.get_metrics(monitor_uid=monitor_uid, deployment_uid='credit')\nprint(custom_metrics)"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "custom_metrics_pandas = subscription.monitoring.get_table_content(monitor_uid=monitor_uid)\n\n%matplotlib inline\ncustom_metrics_pandas.plot.barh(x='id', y='value');"
        }, 
        {
            "source": "### Authors\nLukasz Cmielowski, PhD, is an Automation Architect and Data Scientist at IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge.\n\nWojciech Sobala, Data Scientist at IBM", 
            "cell_type": "markdown", 
            "metadata": {}
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.5", 
            "name": "python3", 
            "language": "python"
        }, 
        "language_info": {
            "mimetype": "text/x-python", 
            "nbconvert_exporter": "python", 
            "version": "3.5.5", 
            "name": "python", 
            "file_extension": ".py", 
            "pygments_lexer": "ipython3", 
            "codemirror_mode": {
                "version": 3, 
                "name": "ipython"
            }
        }
    }, 
    "nbformat": 4
}