quickstart/latest/Fiddler_Quickstart_Simple_Monitoring.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sF3cOBoVx9Ji"
   },
   "source": [
    "# Fiddler Simple Monitoring Quick Start Guide\n",
    "\n",
    "Fiddler is the pioneer in enterprise Model Performance Management (MPM), offering a unified platform that enables Data Science, MLOps, Risk, Compliance, Analytics, and other LOB teams to **monitor, analyze, and improve ML deployments at enterprise scale**.\n",
    "Obtain contextual insights at any stage of the ML lifecycle, improve predictions, increase transparency and fairness, and optimize business revenue.\n",
    "\n",
    "---\n",
    "\n",
    "You can start using Fiddler ***in minutes*** by following these quick steps:\n",
    "\n",
    "1. Connect to Fiddler\n",
    "2. Load a Data Sample\n",
    "3. Define the Model Specifications\n",
    "4. Set the Model Task\n",
    "5. Create a Model\n",
    "6. Set Up Alerts **(Optional)**\n",
    "7. Create a Custom Metric **(Optional)**\n",
    "8. Create a Segment **(Optional)**\n",
    "9. Publish a Pre-production Baseline **(Optional)**\n",
    "10. Configure a Rolling Baseline **(Optional)**\n",
    "11. Publish Production Events"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wUZi_5s7wsGA"
   },
   "source": [
    "# 0. Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Pg-BdRJ4w3LM"
   },
   "outputs": [],
   "source": [
    "%pip install -q fiddler-client\n",
    "\n",
    "import time as time\n",
    "\n",
    "import pandas as pd\n",
    "import fiddler as fdl\n",
    "\n",
    "print(f'Running Fiddler Python client version {fdl.__version__}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hcP0yWfV1GoZ"
   },
   "source": [
    "## 1. Connect to Fiddler\n",
    "\n",
    "Before you can add information about your model with Fiddler, you'll need to connect using the Fiddler Python client.\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "**We need a couple pieces of information to get started.**\n",
    "1. The URL you're using to connect to Fiddler\n",
    "2. Your authorization token\n",
    "\n",
    "Your authorization token can be found by navigating to the **Credentials** tab on the **Settings** page of your Fiddler environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CWErNsofAz6B"
   },
   "outputs": [],
   "source": [
    "URL = ''  # Make sure to include the full URL (including https:// e.g. 'https://your_company_name.fiddler.ai').\n",
    "TOKEN = ''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Constants for this example notebook, change as needed to create your own versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT_NAME = 'quickstart_examples'  # If the project already exists, the notebook will create the model under the existing project.\n",
    "MODEL_NAME = 'bank_churn_simple_monitoring'\n",
    "\n",
    "STATIC_BASELINE_NAME = 'baseline_dataset'\n",
    "ROLLING_BASELINE_NAME = 'rolling_baseline_1week'\n",
    "\n",
    "# Sample data hosted on GitHub\n",
    "PATH_TO_SAMPLE_CSV = 'https://raw.githubusercontent.com/fiddler-labs/fiddler-examples/main/quickstart/data/v3/churn_data_sample.csv'\n",
    "PATH_TO_EVENTS_CSV = 'https://raw.githubusercontent.com/fiddler-labs/fiddler-examples/main/quickstart/data/v3/churn_production_data.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4EttApX9BBIn"
   },
   "source": [
    "Now just run the following to connect to your Fiddler environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "g6ONUHliBAsH"
   },
   "outputs": [],
   "source": [
    "fdl.init(url=URL, token=TOKEN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QpUHeE3dBTHD"
   },
   "source": [
    "#### 1.a Create New or Load Existing Project\n",
    "\n",
    "Once you connect, you can create a new project by specifying a unique project name in the fld.Project constructor and calling the `create()` method. If the project already exists, it will load it for use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vh46Mp1MBSMj"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    # Create project\n",
    "    project = fdl.Project(name=PROJECT_NAME).create()\n",
    "    print(f'New project created with id = {project.id} and name = {project.name}')\n",
    "except fdl.Conflict:\n",
    "    # Get project by name\n",
    "    project = fdl.Project.from_name(name=PROJECT_NAME)\n",
    "    print(f'Loaded existing project with id = {project.id} and name = {project.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tlp8zEvlqO8I"
   },
   "source": [
    "You should now be able to see the newly created project in the Fiddler UI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NygDk2Pyjos4"
   },
   "source": [
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <img src=\"https://raw.githubusercontent.com/fiddler-labs/fiddler-examples/main/quickstart/images/simple_monitoring_1.png\" />\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "V3V2EhIdwA_Q"
   },
   "source": [
    "## 2. Load a Data Sample\n",
    "\n",
    "In this example, we'll be considering the case where we're a bank and we have **a model that predicts churn for our customers**.\n",
    "  \n",
    "In order to get insights into the model's performance, **Fiddler needs a small sample of data** to learn the schema of incoming data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TFVJcU8uCY7z"
   },
   "outputs": [],
   "source": [
    "sample_data_df = pd.read_csv(PATH_TO_SAMPLE_CSV)\n",
    "column_list  = sample_data_df.columns\n",
    "sample_data_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IK3sL73_4FxO"
   },
   "source": [
    "## 3. Define the Model Specifications\n",
    "\n",
    "In order to create a model in Fiddler, create a ModelSpec object with information about what each column of your data sample should used for.\n",
    "\n",
    "Fiddler supports four column types:\n",
    "1. **Inputs**\n",
    "2. **Outputs** (Model predictions)\n",
    "3. **Target** (Ground truth values)\n",
    "4. **Metadata**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_columns = list(\n",
    "    column_list.drop(['predicted_churn', 'churn', 'customer_id', 'timestamp'])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wqyBTZn3NJPU"
   },
   "outputs": [],
   "source": [
    "model_spec = fdl.ModelSpec(\n",
    "    inputs=input_columns,\n",
    "    outputs=['predicted_churn'],\n",
    "    targets=['churn'],  # Note: only a single Target column is allowed, use metadata columns and custom metrics for additional targets\n",
    "    metadata=['customer_id', 'timestamp'],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have columns in your ModelSpec which denote **prediction IDs or timestamps**, then Fiddler can use these to power its analytics accordingly.\n",
    "\n",
    "Let's call them out here and use them when configuring the Model in step 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "id_column = 'customer_id'\n",
    "timestamp_column = 'timestamp'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Set the Model Task\n",
    "\n",
    "Fiddler supports a variety of model tasks. In this case, we're adding a binary classification model.\n",
    "\n",
    "For this, we'll create a ModelTask object and an additional ModelTaskParams object to specify the ordering of our positive and negative labels.\n",
    "\n",
    "*For a detailed breakdown of all supported model tasks, click [here](https://docs.fiddler.ai/product-guide/task-types).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_task = fdl.ModelTask.BINARY_CLASSIFICATION\n",
    "\n",
    "task_params = fdl.ModelTaskParams(target_class_order=['no', 'yes'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Create a Model\n",
    "\n",
    "Create a Model object and publish it to Fiddler, passing in\n",
    "1. Your data sample\n",
    "2. The ModelSpec object\n",
    "3. The ModelTask and ModelTaskParams objects\n",
    "4. The ID and timestamp columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = fdl.Model.from_data(\n",
    "    name=MODEL_NAME,\n",
    "    project_id=project.id,\n",
    "    source=sample_data_df,\n",
    "    spec=model_spec,\n",
    "    task=model_task,\n",
    "    task_params=task_params,\n",
    "    event_id_col=id_column,\n",
    "    event_ts_col=timestamp_column,\n",
    ")\n",
    "\n",
    "model.create()\n",
    "print(f'New model created with id = {model.id} and name = {model.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0GmqFXTYwRg6"
   },
   "source": [
    "On the project page, you should now be able to see the newly onboarded model with its model schema."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wbaDq7wZjos8"
   },
   "source": [
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <img src=\"https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/images/simple_monitoring_3.png?raw=true\" />\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ed1_OS6ajos8"
   },
   "source": [
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <img src=\"https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/images/simple_monitoring_4.png?raw=true\" />\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R-hj33B10Ssw"
   },
   "source": [
    "## 6. Set Up Alerts (Optional)\n",
    "\n",
    "Fiddler allows creating alerting rules when your data or model predictions deviate from expected behavior.\n",
    "\n",
    "The alert rules can compare metrics to **absolute** or **relative** values.\n",
    "\n",
    "Please refer to [our documentation](https://docs.fiddler.ai/client-guide/alerts-with-fiddler-client) for more information on Alert Rules.\n",
    "\n",
    "---\n",
    "  \n",
    "Let's set up some alert rules.\n",
    "\n",
    "The following API call sets up a Data Integrity type rule which triggers an email notification when published events have 2 or more range violations in any 1 day bin for the `numofproducts` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_M9F22O621Ec"
   },
   "outputs": [],
   "source": [
    "alert_rule_1 = fdl.AlertRule(\n",
    "    name='Bank Churn Range Violation Alert',\n",
    "    model_id=model.id,\n",
    "    metric_id='range_violation_count',\n",
    "    bin_size=fdl.BinSize.DAY,\n",
    "    compare_to=fdl.CompareTo.RAW_VALUE,\n",
    "    priority=fdl.Priority.HIGH,\n",
    "    warning_threshold=2,\n",
    "    critical_threshold=3,\n",
    "    condition=fdl.AlertCondition.GREATER,\n",
    "    columns=['numofproducts'],\n",
    ")\n",
    "\n",
    "alert_rule_1.create()\n",
    "print(\n",
    "    f'New alert rule created with id = {alert_rule_1.id} and name = {alert_rule_1.name}'\n",
    ")\n",
    "\n",
    "# Set notification configuration for the alert rule, a single email address for this simple example\n",
    "alert_rule_1.set_notification_config(emails=['name@google.com'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xb1Zr4zj6sxN"
   },
   "source": [
    "Let's add a second alert rule.\n",
    "\n",
    "This one sets up a Performance type rule which triggers an email notification when precision metric is 5% higher than that from 1 hr bin one day ago."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alert_rule_2 = fdl.AlertRule(\n",
    "    name='Bank Churn Performance Alert',\n",
    "    model_id=model.id,\n",
    "    metric_id='precision',\n",
    "    bin_size=fdl.BinSize.HOUR,\n",
    "    compare_to=fdl.CompareTo.TIME_PERIOD,\n",
    "    compare_bin_delta=24,  # Multiple of the bin size\n",
    "    condition=fdl.AlertCondition.GREATER,\n",
    "    warning_threshold=0.05,\n",
    "    critical_threshold=0.1,\n",
    "    priority=fdl.Priority.HIGH,\n",
    ")\n",
    "\n",
    "alert_rule_2.create()\n",
    "print(\n",
    "    f'New alert rule created with id = {alert_rule_2.id} and name = {alert_rule_2.name}'\n",
    ")\n",
    "\n",
    "# Set notification configuration for the alert rule, a single email address for this simple example\n",
    "alert_rule_2.set_notification_config(emails=['name@google.com'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GZNJ5NNm5vnA"
   },
   "source": [
    "## 7. Create a Custom Metric (Optional)\n",
    "\n",
    "Fiddler's [Custom Metric API](https://docs.fiddler.ai/python-client-3-x/api-methods-30#custommetric) enables user-defined formulas for custom metrics.  Custom metrics will be tracked over time and can be used in Charts and Alerts just like the many out of the box metrics provided by Fiddler.  Custom metrics can also be managed in the Fiddler UI.\n",
    "\n",
    "Please refer to [our documentation](https://docs.fiddler.ai/product-guide/monitoring-platform/custom-metrics) for more information on Custom Metrics.\n",
    "\n",
    "---\n",
    "  \n",
    "Let's create an example custom metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "J7uza2kwjos9"
   },
   "outputs": [],
   "source": [
    "custom_metric = fdl.CustomMetric(\n",
    "    name='Lost Revenue',\n",
    "    model_id=model.id,\n",
    "    description='A metric to track revenue lost for each false positive prediction.',\n",
    "    definition=\"\"\"sum(if(fp(),1,0) * -100)\"\"\",  # This is an excel like formula which adds -$100 for each false positive predicted by the model\n",
    ")\n",
    "\n",
    "custom_metric.create()\n",
    "print(\n",
    "    f'New custom metric created with id = {custom_metric.id} and name = {custom_metric.name}'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z47Kx5Iujos9"
   },
   "source": [
    "## 8. Create a Segment (Optional)\n",
    "Fiddler's [Segment API](https://docs.fiddler.ai/python-client-3-x/api-methods-30#segments) enables defining named cohorts/sub-segments in your production data. These segments can be tracked over time, added to charts, and alerted upon. Segments can also be managed in the Fiddler UI.\n",
    "\n",
    "Please refer to our [documentation](https://docs.fiddler.ai/product-guide/monitoring-platform/segments) for more information on the creation and management of segments.\n",
    "\n",
    "Let's create a segment to track customers from Hawaii for a specific age range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oZ1iE8wqjos-"
   },
   "outputs": [],
   "source": [
    "segment = fdl.Segment(\n",
    "    name='Hawaii Customers between 30 and 60',\n",
    "    model_id=model.id,\n",
    "    description='Hawaii Customers between 30 and 60',\n",
    "    definition=\"(age<60 and age>30) and geography=='Hawaii'\",\n",
    ")\n",
    "\n",
    "segment.create()\n",
    "print(f'New segment created with id = {segment.id} and name = {segment.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Publish a Static Baseline (Optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since Fiddler already knows how to process data for your model, we can now add a **baseline dataset**.\n",
    "\n",
    "You can think of this as a static dataset which represents **\"golden data,\"** or the kind of data your model expects to receive.\n",
    "\n",
    "Then, once we start sending production data to Fiddler, you'll be able to see **drift scores** telling you whenever it starts to diverge from this static baseline.\n",
    "\n",
    "***\n",
    "\n",
    "Let's publish our **original data sample** as a pre-production dataset. This will automatically add it as a baseline for the model.\n",
    "\n",
    "\n",
    "*For more information on how to design your baseline dataset, [click here](https://docs.fiddler.ai/client-guide/creating-a-baseline-dataset).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_publish_job = model.publish(\n",
    "    source=sample_data_df,\n",
    "    environment=fdl.EnvType.PRE_PRODUCTION,\n",
    "    dataset_name=STATIC_BASELINE_NAME,\n",
    ")\n",
    "print(\n",
    "    f'Initiated pre-production environment data upload with Job ID = {baseline_publish_job.id}'\n",
    ")\n",
    "\n",
    "# Uncomment the line below to wait for the job to finish, otherwise it will run in the background.\n",
    "# You can check the status on the Jobs page in the Fiddler UI or use the job ID to query the job status via the API.\n",
    "# baseline_publish_job.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P21mz652jos9"
   },
   "source": [
    "## 10. Configure a Rolling Baseline (Optional)\n",
    "\n",
    "Fiddler also allows you to configure a baseline based on **past production data.**\n",
    "\n",
    "This means instead of looking at a static slice of data, it will look into past production events and use what it finds for drift calculation.\n",
    "\n",
    "Please refer to [our documentation](https://docs.fiddler.ai/client-guide/creating-a-baseline-dataset) for more information on Baselines.\n",
    "\n",
    "---\n",
    "  \n",
    "Let's set up a rolling baseline that will allow us to calculate drift relative to production data from 1 week back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "b2Fv0N3tjos9"
   },
   "outputs": [],
   "source": [
    "rolling_baseline = fdl.Baseline(\n",
    "    model_id=model.id,\n",
    "    name=ROLLING_BASELINE_NAME,\n",
    "    type_=fdl.BaselineType.ROLLING,\n",
    "    environment=fdl.EnvType.PRODUCTION,\n",
    "    window_bin_size=fdl.WindowBinSize.DAY,  # Size of the sliding window\n",
    "    offset_delta=7,  # How far back to set our window (multiple of window_bin_size)\n",
    ")\n",
    "\n",
    "rolling_baseline.create()\n",
    "print(\n",
    "    f'New rolling baseline created with id = {rolling_baseline.id} and name = {rolling_baseline.name}'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1BKLkIgPjos-"
   },
   "source": [
    "## 11. Publish Production Events\n",
    "\n",
    "Finally, let's send in some production data!\n",
    "\n",
    "\n",
    "Fiddler will **monitor this data and compare it to your baseline to generate powerful insights into how your model is behaving**.\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "Each record sent to Fiddler is called **an event**.\n",
    "  \n",
    "Let's load some sample events from a CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1dmA8PP7GTdJ"
   },
   "outputs": [],
   "source": [
    "production_data_df = pd.read_csv(PATH_TO_EVENTS_CSV)\n",
    "\n",
    "# Shift the timestamps of the production events to be as recent as today\n",
    "production_data_df['timestamp'] = production_data_df['timestamp'] + (\n",
    "    int(time.time() * 1000) - production_data_df['timestamp'].max()\n",
    ")\n",
    "production_data_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TI-gqrMmhEOe"
   },
   "outputs": [],
   "source": [
    "production_publish_job = model.publish(production_data_df)\n",
    "\n",
    "print(\n",
    "    f'Initiated production environment data upload with Job ID = {production_publish_job.id}'\n",
    ")\n",
    "\n",
    "# Uncomment the line below to wait for the job to finish, otherwise it will run in the background.\n",
    "# You can check the status on the Jobs page in the Fiddler UI or use the job ID to query the job status via the API.\n",
    "# production_publish_job.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vR6IR8YNHgG1"
   },
   "source": [
    "# Get Insights\n",
    "  \n",
    "Return to your Fiddler environment to get enhanced observability into your model's performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-DZMVHAEBzC_"
   },
   "source": [
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <img src=\"https://raw.githubusercontent.com/fiddler-labs/fiddler-examples/main/quickstart/images/simple_monitoring_5.png\" />\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Jdkj1eHgOTAO"
   },
   "source": [
    "**What's Next?**\n",
    "\n",
    "Try the [LLM Monitoring - Quick Start Notebook](https://docs.fiddler.ai/quickstart-notebooks/simple-llm-monitoring)\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "**Questions?**  \n",
    "  \n",
    "Check out [our docs](https://docs.fiddler.ai/) for a more detailed explanation of what Fiddler has to offer.\n",
    "\n",
    "Join our [community Slack](http://fiddler-community.slack.com/) to ask any questions!\n",
    "\n",
    "If you're still looking for answers, fill out a ticket on [our support page](https://fiddlerlabs.zendesk.com/) and we'll get back to you shortly."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}