quickstart/latest/Fiddler_Quickstart_NLP_Multiclass_Monitoring.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f0c3549a",
   "metadata": {
    "id": "f0c3549a"
   },
   "source": [
    "# Monitoring a Multiclass Classifier Model with Text Inputs\n",
    "Unstructured data such as text are usually represented as high-dimensional vectors when processed by ML models. In this example notebook we present how [Fiddler Vector Monitoring](https://www.fiddler.ai/blog/monitoring-natural-language-processing-and-computer-vision-models-part-1) can be used to monitor NLP models using a text classification use case.\n",
    "\n",
    "Following the steps in this notebook you can see how to onboard models that deal with unstructured text inputs. In this example, we use the 20Newsgroups dataset and train a multi-class classifier that is applied to vector embeddings of text documents.\n",
    "\n",
    "We monitor this model at production time and assess the performance of Fiddler's vector monitoring by manufacturing synthetic drift via sampling from specific text categories at different deployment time intervals.\n",
    "\n",
    "---\n",
    "\n",
    "Now we perform the following steps to demonstrate how Fiddler NLP monitoring works:\n",
    "\n",
    "1. Connect to Fiddler\n",
    "2. Load a Data Sample\n",
    "3. Define the Model Specifications\n",
    "4. Create a Model\n",
    "5. Publish a Static Baseline (Optional)\n",
    "6. Publish Production Events\n",
    "\n",
    "Get insights!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6IvnT5Rpwpcy",
   "metadata": {
    "id": "6IvnT5Rpwpcy"
   },
   "source": [
    "## 0. Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "380bfb82",
   "metadata": {
    "id": "380bfb82"
   },
   "outputs": [],
   "source": [
    "\n",
    "%pip install -q fiddler-client\n",
    "\n",
    "import time as time\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import fiddler as fdl\n",
    "\n",
    "print(f\"Running Fiddler Python client version {fdl.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "buFkZ_xkZgyN",
   "metadata": {
    "id": "buFkZ_xkZgyN"
   },
   "source": [
    "## 1. Connect to Fiddler\n",
    "\n",
    "Before you can add information about your model with Fiddler, you'll need to connect using the Fiddler Python client.\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "**We need a couple pieces of information to get started.**\n",
    "1. The URL you're using to connect to Fiddler\n",
    "2. Your authorization token\n",
    "\n",
    "Your authorization token can be found by navigating to the **Credentials** tab on the **Settings** page of your Fiddler environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a41e2e8d",
   "metadata": {
    "id": "a41e2e8d"
   },
   "outputs": [],
   "source": [
    "URL = ''  # Make sure to include the full URL (including https:// e.g. 'https://your_company_name.fiddler.ai').\n",
    "TOKEN = ''"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13a8d868",
   "metadata": {},
   "source": [
    "Constants for this example notebook, change as needed to create your own versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a962e21b",
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT_NAME = 'quickstart_examples'\n",
    "MODEL_NAME = 'nlp_newsgroups_multiclass'\n",
    "\n",
    "STATIC_BASELINE_NAME = 'baseline_dataset'\n",
    "\n",
    "# Sample data hosted on GitHub\n",
    "PATH_TO_SAMPLE_CSV = 'https://raw.githubusercontent.com/fiddler-labs/fiddler-examples/main/quickstart/data/v3/nlp_text_multiclassifier_data_sample.csv'\n",
    "PATH_TO_EVENTS_CSV = 'https://media.githubusercontent.com/media/fiddler-labs/fiddler-examples/main/quickstart/data/v3/nlp_text_multiclassifier_production_data.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "024b0fda",
   "metadata": {
    "id": "024b0fda"
   },
   "source": [
    "Now just run the following to connect to your Fiddler environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "627e583c",
   "metadata": {
    "id": "627e583c"
   },
   "outputs": [],
   "source": [
    "fdl.init(url=URL, token=TOKEN)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0099d99-a241-406b-97ba-05c8da60d924",
   "metadata": {
    "id": "d0099d99-a241-406b-97ba-05c8da60d924"
   },
   "source": [
    "#### 1.a Create New or Load Existing Project\n",
    "\n",
    "Once you connect, you can create a new project by specifying a unique project name in the fld.Project constructor and calling the `create()` method. If the project already exists, it will load it for use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "CWukS_D9aEU4",
   "metadata": {
    "id": "CWukS_D9aEU4"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    # Create project\n",
    "    project = fdl.Project(name=PROJECT_NAME).create()\n",
    "    print(f'New project created with id = {project.id} and name = {project.name}')\n",
    "except fdl.Conflict:\n",
    "    # Get project by name\n",
    "    project = fdl.Project.from_name(name=PROJECT_NAME)\n",
    "    print(f'Loaded existing project with id = {project.id} and name = {project.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1884a477",
   "metadata": {
    "id": "1884a477"
   },
   "source": [
    "## 2. Load a Data Sample"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "kLTJ_yQ6z7Kk",
   "metadata": {
    "id": "kLTJ_yQ6z7Kk"
   },
   "source": [
    "Now we retrieve the 20Newsgroup dataset. This dataset is fetched from the [scikit-learn real-world datasets](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_20newsgroups.html#) and pre-processed using [this notebook](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/pre-proccessing/20newsgroups_prep_vectorization.ipynb).  For simplicity sake, we have stored a copy in our GitHub repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d384c6e-8873-4b61-92c2-4ec7ae9bd4c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_data_df = pd.read_csv(PATH_TO_SAMPLE_CSV)\n",
    "sample_data_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "roikR8aTmfNE",
   "metadata": {
    "id": "roikR8aTmfNE"
   },
   "source": [
    "## 3. Define the Model Specifications"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1Ex_5KxtiHs5",
   "metadata": {
    "id": "1Ex_5KxtiHs5"
   },
   "source": [
    "Next we should tell Fiddler a bit more about the model by creating a `ModelSpec` object that specifies the model's inputs, outputs, targets, and metadata, along with other information such as the enrichments we want performed on our model's data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bIWeThOD0XVh",
   "metadata": {
    "id": "bIWeThOD0XVh"
   },
   "source": [
    "### Instruct Fiddler to generate embeddings for our unstructured model input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "QJv_IGSb0jCS",
   "metadata": {
    "id": "QJv_IGSb0jCS"
   },
   "source": [
    "Fiddler offers a powerful set of enrichment services that we can use to enhance how we monitor our model's performance.  In this example, we instruct Fiddler to generate embeddings for our unstructured text.  These generated embeddings are a numerical vector that represent the content and context of our unstructured input field, _original_text_.  These embeddings then power Fiddler's vector monitoring capability for detecting drift.\n",
    "\n",
    "Before creating a `ModelSpec` object, we define a custom feature using the [fdl.Enrichment()](https://docs.fiddler.ai/python-client-3-x/api-methods-30#fdl.enrichment-private-preview) API. When creating an enrichment, a name must be assigned to the custom feature using the `name` argument. Each enrichment appears in the monitoring tab in the Fiddler UI with this assigned name. Finally, the default clustering setup can be modified by passing the number of cluster centroids to the `n_clusters` argument.\n",
    "\n",
    "Here we define an [embedding fdl.Enrichment](https://docs.fiddler.ai/python-client-3-x/api-methods-30#embedding-private-preview) and then use that embedding enrichment to create a [fdl.TextEmbedding](https://docs.fiddler.ai/python-client-3-x/api-methods-30#customfeature) input that can be used to track drift and to be plotted in Fiddler's UMAP visualizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dlUZbFLya6rv",
   "metadata": {
    "id": "dlUZbFLya6rv"
   },
   "outputs": [],
   "source": [
    "fiddler_backend_enrichments = [\n",
    "    fdl.Enrichment(\n",
    "        name='Enrichment Text Embedding',\n",
    "        enrichment='embedding',\n",
    "        columns=['original_text'],\n",
    "    ),\n",
    "    fdl.TextEmbedding(\n",
    "        name='Original TextEmbedding',\n",
    "        source_column='original_text',\n",
    "        column='Enrichment Text Embedding',\n",
    "        n_clusters=6,\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2632df25-8f0e-42c8-97f9-4268a2ea6604",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_spec = fdl.ModelSpec(\n",
    "    inputs=['original_text'],\n",
    "    outputs=[col for col in sample_data_df.columns if col.startswith('prob_')],\n",
    "    targets=['target'],\n",
    "    metadata=['n_tokens', 'string_size', 'timestamp'],\n",
    "    custom_features=fiddler_backend_enrichments,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30b0e718",
   "metadata": {},
   "source": [
    "If you have columns in your ModelSpec which denote **prediction IDs or timestamps**, then Fiddler can use these to power its analytics accordingly. Here we are just noting the column with the event datetime."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d55492df-924f-4d19-95ac-eda0f3ee70d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "timestamp_column = 'timestamp'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a216974f",
   "metadata": {},
   "source": [
    "Fiddler supports a variety of model tasks. In this case, we're setting the task type to multi-class classification to inform Fiddler that is the type of model we are monitoring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35ca3a11-8693-400e-a7a0-fb9d865615c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_task = fdl.ModelTask.MULTICLASS_CLASSIFICATION\n",
    "\n",
    "task_params = fdl.ModelTaskParams(\n",
    "    target_class_order=[col[5:] for col in model_spec.outputs]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e227383b",
   "metadata": {},
   "source": [
    "## 4. Create a Model\n",
    "\n",
    "Once we've specified information about our model, we can onboard (add) it to Fiddler by calling `Model.create`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4b691a7-d05c-40ba-91e1-d0af374461dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = fdl.Model.from_data(\n",
    "    name=MODEL_NAME,\n",
    "    project_id=project.id,\n",
    "    source=sample_data_df,\n",
    "    spec=model_spec,\n",
    "    task=model_task,\n",
    "    task_params=task_params,\n",
    "    event_ts_col=timestamp_column,\n",
    ")\n",
    "\n",
    "model.create()\n",
    "print(f'New model created with id = {model.id} and name = {model.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "891cc574",
   "metadata": {},
   "source": [
    "## 5. Publish a Static Baseline (Optional)\n",
    "\n",
    "Since Fiddler already knows how to process data for your model, we can now add a **baseline dataset**.\n",
    "\n",
    "You can think of this as a static dataset which represents **\"golden data,\"** or the kind of data your model expects to receive. \n",
    "\n",
    "Note : The data given during the model creation is purely for schema inference and not a default baseline.\n",
    "\n",
    "Then, once we start sending production data to Fiddler, you'll be able to see **drift scores** telling you whenever it starts to diverge from this static baseline.\n",
    "\n",
    "***\n",
    "\n",
    "Let's publish our **original data sample** as a pre-production dataset. This will automatically add it as a baseline for the model.\n",
    "\n",
    "\n",
    "*For more information on how to design your baseline dataset, [click here](https://docs.fiddler.ai/client-guide/creating-a-baseline-dataset).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1bdba54",
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_publish_job = model.publish(\n",
    "    source=sample_data_df,\n",
    "    environment=fdl.EnvType.PRE_PRODUCTION,\n",
    "    dataset_name=STATIC_BASELINE_NAME,\n",
    ")\n",
    "print(\n",
    "    f'Initiated pre-production environment data upload with Job ID = {baseline_publish_job.id}'\n",
    ")\n",
    "\n",
    "# Uncomment the line below to wait for the job to finish, otherwise it will run in the background.\n",
    "# You can check the status on the Jobs page in the Fiddler UI or use the job ID to query the job status via the API.\n",
    "# baseline_publish_job.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "L6L9yGBdcVFU",
   "metadata": {
    "id": "L6L9yGBdcVFU"
   },
   "source": [
    "## 6. Publish Production Events"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "uWB1dDYPnRSf",
   "metadata": {
    "id": "uWB1dDYPnRSf"
   },
   "source": [
    "Now let's publish some sample production events into Fiddler. For the timestamps in the middle of the event dataset, we've over-sampled from certain topics which introduces synthetic drift. This oversampling to inject drift allows us to better illustrate how Fiddler's vector monitoring approach detects drift in our unstructured inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2363c914-6c61-40ac-8c0f-1f59114719f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "production_data_df = pd.read_csv(PATH_TO_EVENTS_CSV)\n",
    "\n",
    "# Shift the timestamps of the production events to be as recent as today\n",
    "production_data_df['timestamp'] = production_data_df['timestamp'] + (\n",
    "    int(time.time() * 1000) - production_data_df['timestamp'].max()\n",
    ")\n",
    "production_data_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7QrQ_2KMdKZR",
   "metadata": {
    "id": "7QrQ_2KMdKZR"
   },
   "source": [
    "Next, let's publish events our events with the synthetic drift"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "197GcBxRdKGx",
   "metadata": {
    "id": "197GcBxRdKGx"
   },
   "outputs": [],
   "source": [
    "production_publish_job = model.publish(production_data_df)\n",
    "\n",
    "print(\n",
    "    f'Initiated production environment data upload with Job ID = {production_publish_job.id}'\n",
    ")\n",
    "\n",
    "# Uncomment the line below to wait for the job to finish, otherwise it will run in the background.\n",
    "# You can check the status on the Jobs page in the Fiddler UI or use the job ID to query the job status via the API.\n",
    "# production_publish_job.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbf936e3-407b-4f1e-9374-3072930c4dc4",
   "metadata": {
    "id": "fbf936e3-407b-4f1e-9374-3072930c4dc4"
   },
   "source": [
    "# Get Insights\n",
    "\n",
    "\n",
    "**You're all done!**\n",
    "  \n",
    "Head to the Fiddler UI to start getting enhanced observability into your model's performance.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qeV8BFLe9Pby",
   "metadata": {
    "id": "qeV8BFLe9Pby"
   },
   "source": [
    "In particular, you can go to your model's default dashboard in Fiddler and check out the resulting drift chart . The first image below is a screenshot of the automatically generated drift chart which shows the drift of the original text. The second image is a custom drift chart with traffic details at an hourly time bin. (Annotation bubbles are not generated by the Fiddler UI.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "325aadc1",
   "metadata": {},
   "source": [
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <img src=\"https://raw.githubusercontent.com/fiddler-labs/fiddler-examples/main/quickstart/images/nlp_multiiclass_drift_2.png\"  />\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9035d78",
   "metadata": {},
   "source": [
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <img src=\"https://raw.githubusercontent.com/fiddler-labs/fiddler-examples/main/quickstart/images/nlp_multiiclass_drift.png\"  />\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}