eval-report.ml10m.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a9a7d17a",
   "metadata": {
    "papermill": {
     "duration": 0.027455,
     "end_time": "2022-02-17T22:25:40.424875",
     "exception": false,
     "start_time": "2022-02-17T22:25:40.397420",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Evaluation Analysis for Recommender Output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3716c02",
   "metadata": {
    "papermill": {
     "duration": 0.020054,
     "end_time": "2022-02-17T22:25:40.466075",
     "exception": false,
     "start_time": "2022-02-17T22:25:40.446021",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this section we will analyze and compare the generated recommendations and predictions from a predefined list of algorithms with the goal of assessing the performance of each algorithm with respect to a metric. In other words, we would rank the algorithms for each metric considered with respect to performance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5e1d5c5",
   "metadata": {
    "papermill": {
     "duration": 0.030108,
     "end_time": "2022-02-17T22:25:40.516185",
     "exception": false,
     "start_time": "2022-02-17T22:25:40.486077",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "499cbd25",
   "metadata": {
    "papermill": {
     "duration": 0.020534,
     "end_time": "2022-02-17T22:25:40.557158",
     "exception": false,
     "start_time": "2022-02-17T22:25:40.536624",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Below are the list of packages required to successfully run the analysis. They are divided into partitions to signify their specific task.<br>\n",
    "We need the pathlib package for working with files and folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "055ac6c3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:25:40.602906Z",
     "iopub.status.busy": "2022-02-17T22:25:40.602734Z",
     "iopub.status.idle": "2022-02-17T22:25:40.608825Z",
     "shell.execute_reply": "2022-02-17T22:25:40.608360Z"
    },
    "papermill": {
     "duration": 0.031846,
     "end_time": "2022-02-17T22:25:40.610218",
     "exception": false,
     "start_time": "2022-02-17T22:25:40.578372",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24325944",
   "metadata": {
    "papermill": {
     "duration": 0.020224,
     "end_time": "2022-02-17T22:25:40.651423",
     "exception": false,
     "start_time": "2022-02-17T22:25:40.631199",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We would use the pandas for analyzing and manipulating our data while seaborn and matplotlib are used for data visualization. statsmodels.graphics.gofplots and scipy.stats.shapiro are used for normality check. Scipy.stats.friedmanchisquare is a non-parametric test used to determine the statistical significance in metric results and the wilcoxon test is used for pairwise comparison of sample data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f274b2e4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:25:40.692776Z",
     "iopub.status.busy": "2022-02-17T22:25:40.692433Z",
     "iopub.status.idle": "2022-02-17T22:25:45.422048Z",
     "shell.execute_reply": "2022-02-17T22:25:45.421460Z"
    },
    "papermill": {
     "duration": 4.752199,
     "end_time": "2022-02-17T22:25:45.423689",
     "exception": false,
     "start_time": "2022-02-17T22:25:40.671490",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from statsmodels.graphics.gofplots import qqplot\n",
    "from scipy.stats import shapiro\n",
    "from scipy.stats import friedmanchisquare, wilcoxon\n",
    "from itertools import combinations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49e41844",
   "metadata": {
    "papermill": {
     "duration": 0.020233,
     "end_time": "2022-02-17T22:25:45.464654",
     "exception": false,
     "start_time": "2022-02-17T22:25:45.444421",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Logging to show what's happening in LensKit routines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c867a0d6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:25:45.506862Z",
     "iopub.status.busy": "2022-02-17T22:25:45.506543Z",
     "iopub.status.idle": "2022-02-17T22:25:45.525114Z",
     "shell.execute_reply": "2022-02-17T22:25:45.524733Z"
    },
    "papermill": {
     "duration": 0.041666,
     "end_time": "2022-02-17T22:25:45.526541",
     "exception": false,
     "start_time": "2022-02-17T22:25:45.484875",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.util.log notebook logging configured\n"
     ]
    }
   ],
   "source": [
    "from lenskit import util\n",
    "util.log_to_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6d8e7da8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:25:45.568901Z",
     "iopub.status.busy": "2022-02-17T22:25:45.568695Z",
     "iopub.status.idle": "2022-02-17T22:25:45.571021Z",
     "shell.execute_reply": "2022-02-17T22:25:45.570647Z"
    },
    "papermill": {
     "duration": 0.024705,
     "end_time": "2022-02-17T22:25:45.572159",
     "exception": false,
     "start_time": "2022-02-17T22:25:45.547454",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "_log = logging.getLogger('eval-report')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c2684d4",
   "metadata": {
    "papermill": {
     "duration": 0.02136,
     "end_time": "2022-02-17T22:25:45.613922",
     "exception": false,
     "start_time": "2022-02-17T22:25:45.592562",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We will use lenskit for training, running, and evaluating recommender algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "64485990",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:25:45.658219Z",
     "iopub.status.busy": "2022-02-17T22:25:45.658037Z",
     "iopub.status.idle": "2022-02-17T22:25:45.664471Z",
     "shell.execute_reply": "2022-02-17T22:25:45.664123Z"
    },
    "papermill": {
     "duration": 0.031526,
     "end_time": "2022-02-17T22:25:45.665789",
     "exception": false,
     "start_time": "2022-02-17T22:25:45.634263",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from lenskit import topn\n",
    "from lenskit.metrics.predict import rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fa4d68d",
   "metadata": {
    "papermill": {
     "duration": 0.020353,
     "end_time": "2022-02-17T22:25:45.707125",
     "exception": false,
     "start_time": "2022-02-17T22:25:45.686772",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Load Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad8cda08",
   "metadata": {
    "papermill": {
     "duration": 0.020509,
     "end_time": "2022-02-17T22:25:45.748145",
     "exception": false,
     "start_time": "2022-02-17T22:25:45.727636",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We specify the dataset we will use for our analysis and the main directory from where we read the recommendation and prediction files. From the main directory we find all the directories associated with the dataset and then read the recommendation and predictions files from those directories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b310f56d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:25:45.793398Z",
     "iopub.status.busy": "2022-02-17T22:25:45.793208Z",
     "iopub.status.idle": "2022-02-17T22:25:45.795478Z",
     "shell.execute_reply": "2022-02-17T22:25:45.795121Z"
    },
    "papermill": {
     "duration": 0.027625,
     "end_time": "2022-02-17T22:25:45.796579",
     "exception": false,
     "start_time": "2022-02-17T22:25:45.768954",
     "status": "completed"
    },
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "dataset = \"ml100k\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4494cb27",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:25:45.838633Z",
     "iopub.status.busy": "2022-02-17T22:25:45.838453Z",
     "iopub.status.idle": "2022-02-17T22:25:45.840532Z",
     "shell.execute_reply": "2022-02-17T22:25:45.840189Z"
    },
    "papermill": {
     "duration": 0.024594,
     "end_time": "2022-02-17T22:25:45.841715",
     "exception": false,
     "start_time": "2022-02-17T22:25:45.817121",
     "status": "completed"
    },
    "tags": [
     "injected-parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "dataset = \"ml10m\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b2276dac",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:25:45.885089Z",
     "iopub.status.busy": "2022-02-17T22:25:45.884781Z",
     "iopub.status.idle": "2022-02-17T22:25:45.886908Z",
     "shell.execute_reply": "2022-02-17T22:25:45.886548Z"
    },
    "papermill": {
     "duration": 0.025341,
     "end_time": "2022-02-17T22:25:45.888275",
     "exception": false,
     "start_time": "2022-02-17T22:25:45.862934",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "output_root = Path(\"runs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c7202910",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:25:45.989041Z",
     "iopub.status.busy": "2022-02-17T22:25:45.988811Z",
     "iopub.status.idle": "2022-02-17T22:25:46.402184Z",
     "shell.execute_reply": "2022-02-17T22:25:46.401846Z"
    },
    "papermill": {
     "duration": 0.49465,
     "end_time": "2022-02-17T22:25:46.403656",
     "exception": false,
     "start_time": "2022-02-17T22:25:45.909006",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "dirs = [fld for fld in output_root.glob(f'{dataset}-*')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "732137e4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:25:46.447480Z",
     "iopub.status.busy": "2022-02-17T22:25:46.447290Z",
     "iopub.status.idle": "2022-02-17T22:26:13.941223Z",
     "shell.execute_reply": "2022-02-17T22:26:13.940263Z"
    },
    "papermill": {
     "duration": 27.517996,
     "end_time": "2022-02-17T22:26:13.942651",
     "exception": false,
     "start_time": "2022-02-17T22:25:46.424655",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 48910800 entries, 0 to 48910799\n",
      "Data columns (total 5 columns):\n",
      " #   Column     Dtype   \n",
      "---  ------     -----   \n",
      " 0   item       int64   \n",
      " 1   score      float64 \n",
      " 2   user       int64   \n",
      " 3   rank       int64   \n",
      " 4   algorithm  category\n",
      "dtypes: category(1), float64(1), int64(3)\n",
      "memory usage: 1.5 GB\n"
     ]
    }
   ],
   "source": [
    "recs = []\n",
    "for fld in dirs: \n",
    "    for file in fld.glob(\"recs-*\"):\n",
    "        rec = pd.read_csv(file, sep=',')\n",
    "        rec[\"algorithm\"] = fld.name.split(\"-\")[1]\n",
    "        recs.append(rec)\n",
    "\n",
    "recs = pd.concat(recs, ignore_index=True)\n",
    "recs = recs.astype({'algorithm': 'category'})\n",
    "recs.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "05c6c117",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:26:13.987994Z",
     "iopub.status.busy": "2022-02-17T22:26:13.987727Z",
     "iopub.status.idle": "2022-02-17T22:26:14.210856Z",
     "shell.execute_reply": "2022-02-17T22:26:14.210517Z"
    },
    "papermill": {
     "duration": 0.247384,
     "end_time": "2022-02-17T22:26:14.211988",
     "exception": false,
     "start_time": "2022-02-17T22:26:13.964604",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Pop', 'Bias', 'IIE', 'UUE', 'ALS', 'III', 'IALS']\n",
       "Categories (7, object): ['ALS', 'Bias', 'IALS', 'IIE', 'III', 'Pop', 'UUE']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rec_algos = recs['algorithm'].unique()\n",
    "rec_algos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "38e63e35",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:26:14.256397Z",
     "iopub.status.busy": "2022-02-17T22:26:14.256208Z",
     "iopub.status.idle": "2022-02-17T22:26:16.276433Z",
     "shell.execute_reply": "2022-02-17T22:26:16.275983Z"
    },
    "papermill": {
     "duration": 2.043615,
     "end_time": "2022-02-17T22:26:16.277656",
     "exception": false,
     "start_time": "2022-02-17T22:26:14.234041",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 2096340 entries, 0 to 2096339\n",
      "Data columns (total 6 columns):\n",
      " #   Column      Dtype   \n",
      "---  ------      -----   \n",
      " 0   user        int64   \n",
      " 1   item        int64   \n",
      " 2   rating      float64 \n",
      " 3   timestamp   int64   \n",
      " 4   prediction  float64 \n",
      " 5   algorithm   category\n",
      "dtypes: category(1), float64(2), int64(3)\n",
      "memory usage: 82.0 MB\n"
     ]
    }
   ],
   "source": [
    "preds = []\n",
    "for fld in dirs:\n",
    "    for file in fld.glob(\"pred-*\"):\n",
    "        pred = pd.read_csv(file, sep=',')\n",
    "        pred[\"algorithm\"] = fld.name.split(\"-\")[1]\n",
    "        preds.append(pred)\n",
    "\n",
    "preds = pd.concat(preds, ignore_index=True)\n",
    "preds = preds.astype({'algorithm': 'category'})\n",
    "preds.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffc52b1a",
   "metadata": {
    "papermill": {
     "duration": 0.021977,
     "end_time": "2022-02-17T22:26:16.321832",
     "exception": false,
     "start_time": "2022-02-17T22:26:16.299855",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We need to load the test data so that we have the ground truths for computing accuracy "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "62d07fde",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:26:16.366425Z",
     "iopub.status.busy": "2022-02-17T22:26:16.366219Z",
     "iopub.status.idle": "2022-02-17T22:26:16.368956Z",
     "shell.execute_reply": "2022-02-17T22:26:16.368579Z"
    },
    "papermill": {
     "duration": 0.026359,
     "end_time": "2022-02-17T22:26:16.370085",
     "exception": false,
     "start_time": "2022-02-17T22:26:16.343726",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "split_root = Path(\"data-split\")\n",
    "split_dir = split_root / dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "80427e65",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:26:16.414549Z",
     "iopub.status.busy": "2022-02-17T22:26:16.414364Z",
     "iopub.status.idle": "2022-02-17T22:26:16.685472Z",
     "shell.execute_reply": "2022-02-17T22:26:16.685093Z"
    },
    "papermill": {
     "duration": 0.294765,
     "end_time": "2022-02-17T22:26:16.686613",
     "exception": false,
     "start_time": "2022-02-17T22:26:16.391848",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>item</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>part</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>362</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838984885</td>\n",
       "      <td>test-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>355</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838984474</td>\n",
       "      <td>test-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>231</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983392</td>\n",
       "      <td>test-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1</td>\n",
       "      <td>589</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838983778</td>\n",
       "      <td>test-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1</td>\n",
       "      <td>539</td>\n",
       "      <td>5.0</td>\n",
       "      <td>838984068</td>\n",
       "      <td>test-1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       user  item  rating  timestamp    part\n",
       "index                                       \n",
       "8         1   362     5.0  838984885  test-1\n",
       "6         1   355     5.0  838984474  test-1\n",
       "2         1   231     5.0  838983392  test-1\n",
       "19        1   589     5.0  838983778  test-1\n",
       "16        1   539     5.0  838984068  test-1"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = []\n",
    "for file in split_dir.glob(\"test-*.csv.gz\"):\n",
    "    test.append(pd.read_csv(file, sep=',', index_col='index').assign(part=file.stem.replace('.csv', '')))\n",
    "\n",
    "test = pd.concat(test)\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3542d1c1",
   "metadata": {
    "papermill": {
     "duration": 0.02241,
     "end_time": "2022-02-17T22:26:16.731757",
     "exception": false,
     "start_time": "2022-02-17T22:26:16.709347",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Top-N Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b49d053",
   "metadata": {
    "papermill": {
     "duration": 0.022477,
     "end_time": "2022-02-17T22:26:16.776649",
     "exception": false,
     "start_time": "2022-02-17T22:26:16.754172",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The topn.RecListAnalysis class computes top-N metrics for recommendation list and takes care of making sure that the recommendations and ground truths are properly matched. Refer to the documentation for detailed explanation of the purpose for the RecListAnalysis class and how the analysis is done - https://lkpy.lenskit.org/en/stable/evaluation/topn-metrics.html "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "8f52e54a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:26:16.822698Z",
     "iopub.status.busy": "2022-02-17T22:26:16.822479Z",
     "iopub.status.idle": "2022-02-17T22:27:00.591233Z",
     "shell.execute_reply": "2022-02-17T22:27:00.590763Z"
    },
    "papermill": {
     "duration": 43.793547,
     "end_time": "2022-02-17T22:27:00.592642",
     "exception": false,
     "start_time": "2022-02-17T22:26:16.799095",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn analyzing 48910800 recommendations (349390 truth rows)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn using rec key columns ['algorithm', 'user']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn using truth key columns ['user']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn numbering truth lists\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn numbering rec lists\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn collecting metric results\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn measured 489108 lists in 33.23s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   INFO] lenskit.topn filling in missing user info (489108 initial rows)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>level_1</th>\n",
       "      <th>nrecs</th>\n",
       "      <th>precision</th>\n",
       "      <th>recip_rank</th>\n",
       "      <th>ndcg</th>\n",
       "      <th>ntruth</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <th>user</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">ALS</th>\n",
       "      <th>1</th>\n",
       "      <td>279474.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>279475.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>279476.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>279477.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.131508</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>279478.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.037037</td>\n",
       "      <td>0.082002</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 level_1  nrecs  precision  recip_rank      ndcg  ntruth\n",
       "algorithm user                                                          \n",
       "ALS       1     279474.0  100.0       0.00    0.000000  0.000000       5\n",
       "          9     279475.0  100.0       0.00    0.000000  0.000000       5\n",
       "          10    279476.0  100.0       0.00    0.000000  0.000000       5\n",
       "          11    279477.0  100.0       0.02    0.100000  0.131508       5\n",
       "          13    279478.0  100.0       0.01    0.037037  0.082002       5"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rla = topn.RecListAnalysis()\n",
    "\n",
    "rla.add_metric(topn.precision)\n",
    "rla.add_metric(topn.recip_rank)\n",
    "rla.add_metric(topn.ndcg)\n",
    "results = rla.compute(recs, test, include_missing=True)\n",
    "results = results.fillna(0)\n",
    "results.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86c63e76",
   "metadata": {
    "papermill": {
     "duration": 0.024791,
     "end_time": "2022-02-17T22:27:00.642059",
     "exception": false,
     "start_time": "2022-02-17T22:27:00.617268",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We will reshape the 'results' dataframe by stacking the columns to index and then use the bar chart to visualize the performance of our algorithms with respect to the precision, reciprocal rank and ndcg metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "93b2e282",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:27:00.692141Z",
     "iopub.status.busy": "2022-02-17T22:27:00.691906Z",
     "iopub.status.idle": "2022-02-17T22:27:00.803156Z",
     "shell.execute_reply": "2022-02-17T22:27:00.802773Z"
    },
    "papermill": {
     "duration": 0.138086,
     "end_time": "2022-02-17T22:27:00.804446",
     "exception": false,
     "start_time": "2022-02-17T22:27:00.666360",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>algorithm</th>\n",
       "      <th>user</th>\n",
       "      <th>metric</th>\n",
       "      <th>val</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ALS</td>\n",
       "      <td>1</td>\n",
       "      <td>level_1</td>\n",
       "      <td>279474.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ALS</td>\n",
       "      <td>1</td>\n",
       "      <td>precision</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ALS</td>\n",
       "      <td>1</td>\n",
       "      <td>recip_rank</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ALS</td>\n",
       "      <td>1</td>\n",
       "      <td>ndcg</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ALS</td>\n",
       "      <td>9</td>\n",
       "      <td>level_1</td>\n",
       "      <td>279475.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  algorithm  user      metric       val\n",
       "0       ALS     1     level_1  279474.0\n",
       "1       ALS     1   precision       0.0\n",
       "2       ALS     1  recip_rank       0.0\n",
       "3       ALS     1        ndcg       0.0\n",
       "4       ALS     9     level_1  279475.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pltData = (results.drop(columns=['nrecs', 'ntruth']).stack()).reset_index()\n",
    "pltData.columns = ['algorithm', 'user', 'metric', 'val']\n",
    "pltData.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "362e2b2d",
   "metadata": {
    "papermill": {
     "duration": 0.031338,
     "end_time": "2022-02-17T22:27:00.860681",
     "exception": false,
     "start_time": "2022-02-17T22:27:00.829343",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We need to determine if the differences we observe in the performances of the algorithms for the various metrics are statistically significant. To achieve this, we will need to use either a parametric or non-parametric statistical test for comparing the differences. We will consider a parametric test - repeated ANOVA measure cause our sample groups are correlated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "09f1867e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:27:00.911098Z",
     "iopub.status.busy": "2022-02-17T22:27:00.910885Z",
     "iopub.status.idle": "2022-02-17T22:27:13.851339Z",
     "shell.execute_reply": "2022-02-17T22:27:13.850843Z"
    },
    "papermill": {
     "duration": 12.967272,
     "end_time": "2022-02-17T22:27:13.852632",
     "exception": false,
     "start_time": "2022-02-17T22:27:00.885360",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABAUAAADQCAYAAAB/Rdj7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA2a0lEQVR4nO3de5hcVZ3u8e9LQgS5BwKJQISjOSIoBMiEzMQREMEEL0EFBZEEhJPBA6KOTBt1RhCGGcyAHBkumagZErwEBkFajUAmCIhcTEBIuIhkACEhTQIJERSBkN/5Y69KdirV3VXVdemuej/PU09Vrb3WXmtX99pV9au111JEYGZmZmZmZmbtZ4tmN8DMzMzMzMzMmsNBATMzMzMzM7M25aCAmZmZmZmZWZtyUMDMzMzMzMysTTkoYGZmZmZmZtamHBQwMzMzMzMza1MOCpiZ2QaSRks6uoftYyRd2sg29ZWk0yVN7mH7RyRNa2SbzMzM+pOB8P4v6WRJlzWzDa3KQQGrWjNPHpIOk/SzRu5T0j6S7pb0qqSza1m3WT8yGijZryUNjohFEXFWY5u0SRskqaL3roiYERFzetjeGREX9r11Zv3XQPjA3x1J35W0bxPq3UvSQ42u16xJRtOP3/+tvhwUsL4YTXudPFYDZwEXNbshZt1JH2J/lz5EPyTpB5LeL+nXkh6XNDbl20bSLEkLJf1W0iRJQ4DzgE9KekDSJyWdK2mmpFuAOfngmaRtJf2npCWSFkv6eB/bfrKkGyXdJOkxSefkjulRSVcA9wN7SvqH1PbFkr6R28fklPagpKtT2rmFQJ6ksyQ9kvLMzdV7WXr8VkkL0vYFkkam9KskXSrpLklPSDq2L8dq1gSj6Qfv2ZIGV1omIk6LiEcaXa/ZQNIC7//Xp/f/xyVNz207RdLvJd0OjM+l7ybphvR+/6Ckv0np/5Reh/mSfiT/kFcWBwXa2EA+eRQdx2btS+n3Stovl+82SQd3l783EbEyIhYCr9eq7WZ18nbg28D+wD7Ap4D3AGcDX015vgbcGhF/BRwO/BuwJfB14JqIGB0R16S8BwOTIuJTRfX8E7A2It4dEfsDtxY3RNIl6RxRfOtuuP5Y4ESyLzDHSRqT0t8BzImIA9PjUSnvaOBgSe9N/f1rwPsi4gDg8yX2Pw04MLX39BLbL0v17A/8AMj/cjqC7HX8EOCRBdZQA/k9O33g/y9JPwVu6eF9e5Cki3L1fi6l31Y4F0h6WdLFku5XFrgb1kO9t0n6l/Rl4vOSPpw+G/xW0n9L2i3lOze15zZlQb/NgiOS/lcq91d9eS3M6mwgv/+PBj4JvJvsXLWnpBHAN8iCAUcC+RFDlwK3p/f7g4CH03ni48CBwMeAMVhZHDW1twPHAVOBhWw8eXyE7ORxDBtPHp+RtCPwG+C/yU4eYyLiTMjeVMlOHu+JiFckHZarZ8PJI+Xdqbghki4hOzkVm9vL0N7N2ifpv4G5wCeAc9JJ5S0RcZ+kf+kmv1mreDIilgBIehhYEBEhaQmwV8pzFPARbYygbwWM7GZ/nRHxSon09wPHF55ExJriDBHxxQrbPj8iXkhtv57sfPQT4A8RcU+u7UcBv03PtyULEhwAXBcRz6e6V5fY/2LgB5J+kvZb7K/JPkgAXA1Mz237SUSsBx4pfJkwa7CB/J7918D+EbG6h/fhycDeZIG7dZKGltjPNsD9EfElSV8HzgHOLJGvYMeIODR3HOPS+fA0oAP4Usq3Tzqe7YDHJF2ZO9Z3kH2mOCUiHuihLrNmG8jv/wsiYm1q+yPAW4FdgNsiYlVKvwb43yn/+8jOGUTEG8BaSe8Bbiy0OQUirQwOCthAPnkUdNe+a4H5ZB8YPgH8Vy/5zVrFq7nH63PP17PxvC/g4xHxWL6gpENK7O9P3dQjIHpqSBVfHIr3V3ieb4OAf42I/yiq66ze2gN8EHgv2Zeof1JuNFE38vvLv67qpZxZPQzk9+z5uUBdd218PzAjItalOkoF9tYDhV8xvw9c30u91+Qe7wFck34oGAI8mdv284h4FXhV0kqgEPgbBtxIdr58uJe6zJptIL//59v+Rq69vb2vF7fLquDLB6ySk8fodBsZEY92s78+nTwqHGaU3/dm7YuI5cALkvYnG440t4rjMWtVNwOfkyQASQem9JfIfikrxy3kfqEr9WtiRHwx19fyt+5G/xwpaaikrcl+9fx1N23/jKRtU727S9oVWAB8QtLOKX2TXxmVTVC4Z0T8kuwXwh3JRhnk3cXGL0MnAnd2f/hmDTeQ37OLA3ul2thrvSX0lj9f778Dl6UREH9HFowo6O4LyVrgGXLXMpsNcP31/b+Ue4HDJO0saUuykVIFC4DPpvoHSdqe7D37w5K2Sp8RPlhBXW3NQQErR38/eXTXPsgCAR3ADoVfV3rJb9Yuzie7hnCxstm1z0/pvwT2TR/uP9nLPv4Z2EnZ9c0PUvoXgUrdSTZs/wHgxxGxqDhDRNwC/BC4O/1Ceh2wXfoV7wLg9tSebxUVHQR8P5X5LXBJRLxYlOcs4BRJi4GTKD0vgVl/1t/fs3tq4y3A6UqTAnZz+cAWQGGiz09RWeBuB2B5ejylzDKvkQUoJ0sqvq7abCDqr+//m4mIFcC5wN1kl0Hdn9v8eeDw9J5+H7BfZHN/dQIPko0iWkQW2LNeKKLSgKy1Ckl7AT+LiHel51el59flt6Vf7P4f8DdkUfynIuJD6c36ZrITy78C7wRejoiL0v4OA85OebcFLie7fvEN4BsR0duQv57ant93yfalfLuRfQA4PyK+kdK6O54N++ymzuFkJ5ftyX6VeRnYNyL+WO1xmNlGkk4md82zmW00wN+zT2bT+Qy6a+Ngsnk8JpBN6vudiLhM0m2pbYskvQxcQraSwlrgk4XrjUvUu6Fcej4plV0O3AP8VUQcpmx+hfxr8RDZhKLkXtcdyS5J/OeIuLHa18LM6kvSthHxsqQ3A3cAUyPi/t7KtTsHBczMrF9wUMDMeiPp5YgovuzHzAwAST8kW6VgK2B2RPxrk5s0IDgoYGZmZmYDgoMCZma156CAWRFJp7D5dcS/jogzmtEeMzMz656ky9l8IsBvR8R/NqM9ZmYDjYMCZmZmZmZmZm3Kqw8kEyZMCLJlbXzzzbfybgOS+7pvvlV0G7Dc133zraLbgOR+7ptvFd265aBA8vzzzze7CWbWAO7rZu3Bfd2s9bmfm9WGgwJmZmZmZmZmbcpBATMzMzMzM7M25aCAmZmZmZmZWZtyUMDMzMzMzMysTTkoYGZmZmZmZtamBje7AWZm1n90dHTQ1dXF8OHDmT59erObY2bWJz6nWTvz/7+Vy0GBNjf+38fXbd+//tyv67ZvMytfJf18yO+GsMWftuCZF58pu5z7upn1V11dXSxfvrzZzTCrqXLfn/2ebuVyUMDMzMzMBozLvvTTsvO++PyfNtyXW+7Miz9cVbvMzAYqBwXMzGyDeHOwnvXEm6PZTTEzM7M+8Hu6lcsTDZqZ2Qavj3+d1458jdfHv97spliLkTRB0mOSlkqaVmL7JEmLJT0gaZGk95Rb1qw72wzZnm3etCPbDNm+2U0xazi/p1u5PFLAzBpC0gTg28Ag4LsRcWHRdqXtRwN/Bk6OiPslbQXcAbyJ7Jx1XUSck8oMBa4B9gKeAj4REWsackBmVjZJg4DLgSOBZcBCSZ0R8Ugu2wKgMyJC0v7AtcA+ZZY1K2n82z7W7CaYmfV7HilgZnWX+1A/EdgXOEHSvkXZJgKj0m0qcGVKfxV4X0QcAIwGJkgal7ZNAxZExCiyLxT+BdGsfxoLLI2IJyLiNWAuMCmfISJejojCGNdtgCi3rJmZmVXPQQEza4RyPtRPAuZE5h5gR0kj0vOXU54t0y1yZWanx7OBY+p5EGZWtd2BZ3LPl6W0TUj6qKTfAT8HPlNJWTMzM6uOgwJm1gjlfKjvNo+kQZIeAFYC8yPi3pRnt4hYAZDudy1VuaSp6RrlRatWrerrsZhZ5VQibbOZryLihojYhyzAd34lZcF93czMrBoOCphZI5Tzob7bPBHxRkSMBvYAxkp6VyWVR8TMiBgTEWOGDRtWSVEzq41lwJ6553sAz3aXOSLuAN4maZdKyrqvm5mZVc5BATNrhHI+1PeaJyJeBG4DJqSk5ySNAEj3K2vWYjOrpYXAKEl7SxoCHA905jNIenuacBRJBwFDgBfKKWtmZmbVc1DAzBqhnA/1ncBkZcYBayNihaRhknYEkLQ18H7gd7kyU9LjKcCNdT4OM6tCRKwDzgRuBh4Fro2IhyWdLun0lO3jwEPpUqHLgU+mOUVKlm34QZhZWcpYfvTEtPzoYkl3STogt+0pSUsKS5M2tuVm7ctLEppZ3UXEOkmFD/WDgFmFLwRp+wxgHtlyhEvJliQ8JRUfAcxOKxhsQfaF4Gdp24XAtZJOBZ4GjmvUMZlZZSJiHlk/z6fNyD3+JvDNcsuaWf9T5hKiTwKHRsQaSROBmcAhue2HR8TzDWu0mTkoYGaNUcYXggDOKFFuMXBgN/t8ATiiti01MzOzKm1YbQhAUmG1oQ1BgYi4K5f/HrLLBc2siep++UCaNfy3kn6Wng+VNF/S4+l+p1zer6ShRo9J+kAu/eA0lGippEtz1xy+SdI1Kf1eSXvlykxJdTwuaQpmZmZmZlZPlS4heirwi9zzAG6RdJ+kqaUKeJURs9prxJwCnye7BrBgGrAgIkYBC9JzJO1Ldp3xfmSTiF2RhiABXAlMBUalW2GSsVOBNRHxduAS0rBDSUOBc8iGIo0FzskHH8zMzMzMrOYqWUL0cLLP8l/OJY+PiIOAicAZkt672c68yoiV0NHRweTJk+no6Gh2UwakugYFJO0BfBD4bi55EjA7PZ5NthZxIX1uRLwaEU+SXVc8Ns0ovn1E3J2GF88pKlPY13XAEWkUwQfI1jJfHRFrgPlsDCSYmZmZmVntlbWEqKT9yb4fTEqXAgIQEc+m+5XADWQ/7pn1qquri+XLl9PV1dXspgxI9Z5T4P8BHcB2ubTdImIFQJpZfNeUvjvZdUUFheFGr6fHxemFMs+kfa2TtBbYmcqHLpmZmZm1jY6ODrq6uhg+fDjTp09vdnOsdWxYbQhYTjYK+FP5DJJGAtcDJ0XE73Pp2wBbRMRL6fFRwHkNa7n1O5d96adl533x+T9tuC+n3JkXf7jqdrWiugUFJH0IWBkR90k6rJwiJdKih/Rqy+TbOJXssgRGjhxZRhPNzMzMBr7Cr2pmtVTmakNfJ/sR74o0Tdi6iBgD7AbckNIGAz+MiJuacBhmbaeeIwXGAx+RdDSwFbC9pO8Dz0kakUYJjABWpvzdDTdaxqazkuaHIRXKLJM0GNgBWJ3SDysqc1txAyNiJtkyKIwZM6bk9U5mZmZmA8H4fx9fdt4hLw5hC7bgmRefKavcrz/36740zdpIGasNnQacVqLcE8ABdW+gtaRthmy/yb1Vpm5zCkTEVyJij4jYi2zo0K0R8WmgEyisBjAFuDE97gSOTysK7E02oeBv0qUGL0kal+YLmFxUprCvY1MdQRadPErSTmmCwaNSmpmZmZmZmbWQ8W/7GEe982TGv+1jzW7KgFTvOQVKuRC4VtKpwNPAcQBpaNG1ZOuYrgPOiIg3UpnPAlcBW5MtW1JYuuR7wNWSlpKNEDg+7Wu1pPPJrmsCOC8iVtf7wMzMzMwGgnhzsJ71xJs9UNLMrN01JCgQEbeRhu+nGUaP6CbfBcAFJdIXAe8qkf4XUlChxLZZwKxq22ytpZKJSirliUrMzGygeX38681ugpmZ9RN1XZLQzMzMzMzMzPovBwXMzMzMzMzM2pSDAmZmZmZmZmZtykEBMzMzMzMzszbloICZNYSkCZIek7RU0rQS2yXp0rR9saSDUvqekn4p6VFJD0v6fK7MuZKWS3og3Y5u5DGZWfnKOAecmPr+Ykl3STogt+0pSUtSP1/U2JabmZm1tmYsSWhmbUbSIOBy4EhgGbBQUmdEPJLLNhEYlW6HAFem+3XAlyLifknbAfdJmp8re0lEXNSoYzGzypV5DngSODQi1kiaCMwkOwcUHB4Rzzes0WZmZm3CIwXMrBHGAksj4omIeA2YC0wqyjMJmBOZe4AdJY2IiBURcT9ARLwEPArs3sjGm1mf9XoOiIi7ImJNenoPsEeD22hmZtaWHBQws0bYHXgm93wZm3+x7zWPpL2AA4F7c8lnpuHGsyTtVKpySVMlLZK0aNWqVVUegpn1QTnngLxTgV/kngdwi6T7JE3trpD7upmZWeUcFDCzRlCJtKgkj6RtgR8DX4iIP6bkK4G3AaOBFcDFpSqPiJkRMSYixgwbNqzCpptZDZRzDsgySoeTBQW+nEseHxEHkV1mdIak95Yq675uZmZWOQcFzKwRlgF75p7vATxbbh5JW5IFBH4QEdcXMkTEcxHxRkSsB75DNkTZzPqfcs4BSNof+C4wKSJeKKRHxLPpfiVwA+7rZmZmNeOggJk1wkJglKS9JQ0Bjgc6i/J0ApPTKgTjgLURsUKSgO8Bj0bEt/IFJI3IPf0o8FD9DsHM+qDXc4CkkcD1wEkR8ftc+jZpklEkbQMchfu6mZlZzXj1ATOru4hYJ+lM4GZgEDArIh6WdHraPgOYBxwNLAX+DJySio8HTgKWSHogpX01IuYB0yWNJhuG/BTwdw05IBvwOjo66OrqYvjw4UyfPr3ZzWl5ZZ4Dvg7sDFyRxQJZFxFjgN2AG1LaYOCHEXFTEw7DzMysJTkoYGYNkb7EzytKm5F7HMAZJcrdSenrkYmIk2rcTGsTXV1dLF++vNnNaCtlnANOA04rUe4J4IC6N9DMzKxNOShgZmYt47Iv/bSsfC8+/6cN9+WWOfPiD1fdLjMzM7P+ynMKmJlZ29lmyPZs86Yd2WbI9s1uiplZS5E0QdJjkpZKmlZi+4lpKeHFku6SdEC5Zc2sPjxSwMzM2s74t32s2U0wM2s5kgYBlwNHkq06slBSZ0Q8ksv2JHBoRKyRNBGYCRxSZlkzqwOPFDAzMzMzs1oYCyyNiCci4jVgLjApnyEi7oqINenpPWRLlJZV1szqw0EBMzMzMzOrhd2BZ3LPl6W07pwK/KKSspKmSlokadGqVav62FwzAwcFzMzMzMysNkqtFhQlM0qHkwUFvlxJ2YiYGRFjImLMsGHDqm6omW3kOQXMzMzMzKwWlgF75p7vATxbnEnS/sB3gYkR8UIlZc2s9jxSwMzMzMzMamEhMErS3pKGAMcDnfkMkkYC1wMnRcTvKylrZvXhkQIlHPwPc+q6//v+bXJd929mZmZm1mgRsU7SmcDNwCBgVkQ8LOn0tH0G8HVgZ+AKSQDr0uUAJcs25UDM2oyDAmZmZmZmVhMRMQ+YV5Q2I/f4NOC0csuaWf3V7fIBSVtJ+o2kByU9LOkbKX2opPmSHk/3O+XKfEXSUkmPSfpALv1gSUvStkuVwoqS3iTpmpR+r6S9cmWmpDoelzSlXsdpZmZmZmZmNlDVc06BV4H3RcQBwGhggqRxwDRgQUSMAhak50jal+zaof2ACWRDigalfV0JTAVGpduElH4qsCYi3g5cAnwz7WsocA5wCNmap+fkgw9mZmZmZmZmVsegQGReTk+3TLcAJgGzU/ps4Jj0eBIwNyJejYgngaXAWEkjgO0j4u6ICGBOUZnCvq4DjkijCD4AzI+I1RGxBpjPxkCCmZmZmZmZmVHn1QckDZL0ALCS7Ev6vcBuEbECIN3vmrLvDjyTK74spe2eHhenb1ImItYBa8kmLuluX8XtmyppkaRFq1at6sORmpmZWU8kTUiXBy6VNK3E9hMlLU63uyQdUG5ZMzMzq15dgwIR8UZEjCZbZ3SspHf1kF2ldtFDerVl8u2bmWY7HTNs2LAemmZmfVXGFwKlOUOWpi8FB6X0PSX9UtKjaX6Sz+fKdDtHiZn1H+lywMuBicC+wAnpssG8J4FDI2J/4HxgZgVlzczMrEp1DQoURMSLwG1kQ/ifS5cEkO5XpmzLgD1zxfYAnk3pe5RI36SMpMHADsDqHvZlZk1Q5of6iWycN2Qq2VwiAOuAL0XEO4FxwBm5siXnKDGzfmcssDQinoiI14C5ZJcAbhARd6VL/gDuYeN7f69lzczMrHr1XH1gmKQd0+OtgfcDvwM6gcJqAFOAG9PjTuD4tKLA3mRfDH6TLjF4SdK4NF/A5KIyhX0dC9ya5h24GThK0k7pl8OjUpqZNUc5H+onAXPSfCT3ADtKGhERKyLifoCIeAl4lI2XA3U3R4mZ9S9lXdaXcyrwiyrLmpmZWQUG13HfI4DZ6RfCLYBrI+Jnku4GrpV0KvA0cBxARDws6VrgEbJfBs+IiDfSvj4LXAVsTfYhofBB4XvA1ZKWko0QOD7ta7Wk84GFKd95EbG6jsdqZj0r9aH+kDLy7A6sKCSkZUcPBO5NSZvMUSJpV0qQNJVs9AEjR46s+iDMrGplXdYHIOlwsqDAe6oo675uZmZWoboFBSJiMdmH9+L0F4AjuilzAXBBifRFwGbzEUTEX0hBhRLbZgGzKmu1mdVJOR/qe8wjaVvgx8AXIuKPlVQeETNJ1yePGTOm5JcJM6ursi7rk7Q/8F1gYvq8UHZZcF83MzOrRkPmFDCztlfOh/pu80jakiwg8IOIuD6Xp7s5Ssysf1kIjJK0t6QhZCP7OvMZJI0ErgdOiojfV1LWzMzMqueggJk1Qjkf6juByWkVgnHA2nRJgMguFXo0Ir5VokypOUrMrB9JywafSTa/z6NklxQ+LOl0SaenbF8nW1b4CkkPSFrUU9mGH4SZmVmLquecAmZmQPahXlLhQ/0gYFbhC0HaPgOYBxwNLAX+DJySio8HTgKWSHogpX01IuYBF1JijhIz639Sn51XlDYj9/g04LRyy5qZmVltOChgZg1RxheCAM4oUe5OSs830OMcJWZmZmZm1jtfPmBmZmZmZmbWphwUMDMzMzMzM2tTDgqYmZmZmZmZtSkHBczMzMzMaqSjo4PJkyfT0dHR7KY0haQJkh6TtFTStBLb95F0t6RXJZ1dtO0pSUvyK5CYWf15okEzM7MG6OjooKuri+HDhzN9+vRmN8fM6qSrq4vly5c3uxlNIWkQcDlwJLAMWCipMyIeyWVbDZwFHNPNbg6PiOfr2lCzGmil9/VugwKSXgKi1CayicK3r1urzMzMBogLPn1sWfkeWbmWV95Yz+quFWWX+dr3r+tL08ysRsrtswCrV67N7tuzr48FlkbEEwCS5gKTgA1BgYhYCayU9MHmNNGsNlopANhtUCAitmtkQ8zMzMzMbEDbHXgm93wZcEgF5QO4RVIA/xERM2vZOLPe1DMA2J+Df2VfPiBpV2CrwvOIeLouLTKzqj16wa113f87v/a+uu7frJVtNWiLTe7NrDW1eV9XibRSI4+7Mz4ink3fO+ZL+l1E3LFJBdJUYCrAyJEjq2+pWR+1Ul/vNSgg6SPAxcBbgJXAW4FHgf3q2zQzM7PWceDOHoBn1g7avK8vA/bMPd8DeLbcwhHxbLpfKekGsssR7ijKMxOYCTBmzJhKAg5mNdVKfb2csMb5wDjg9xGxN3AE8Ou6tsrMzMzMzAaahcAoSXtLGgIcD3SWU1DSNpK2KzwGjgIeqltLzWyDci4feD0iXpC0haQtIuKXkr5Z95aZmZmZmdmAERHrJJ0J3AwMAmZFxMOSTk/bZ0gaDiwCtgfWS/oCsC+wC3CDJMi+o/wwIm5qwmGYtZ1yggIvStoW+BXwA0krgXX1bZaZmZmZmQ00ETEPmFeUNiP3uIvssoJifwQOqG/rzKyUci4fuAPYEfg8cBPwP8CH69gmMzMzMzMzM2uAcoICIhsCdBuwLXBNRLxQz0aZWeuRNEHSY5KWSppWYrskXZq2L5Z0UG7bLEkrJT1UVOZcScslPZBuRzfiWMyscmWcA/aRdLekVyWdXbTtKUlLUj9f1LhWm5mZtb5egwIR8Y2I2A84g2wFgtsl/XfdW2ZmLUPSIOByYCLZdYMnSNq3KNtEYFS6TQWuzG27CpjQze4viYjR6Tavmzxm1kRlngNWA2cBF3Wzm8NTPx9Tv5aamZm1n0oWVVwJdAEvALvWpzlm1qLGAksj4omIeA2YC0wqyjMJmBOZe4AdJY0ASGsUr25oi82slno9B0TEyohYCLzejAaamZm1q16DApI+K+k2YAHZrKD/JyL2r3fDzKyl7A48k3u+LKVVmqeUM9PlBrMk7VQqg6SpkhZJWrRq1apK2m1mtVFt/y4I4BZJ90ma2l0m93UzM7PKlTNS4K3AFyJiv4g4JyIeqXejzKzlqERaVJGn2JXA24DRwArg4lKZImJmRIyJiDHDhg3rZZdmVgfV9O+88RFxENnlB2dIem+pTO7rZmZmlStnToFpEfFApTuWtKekX0p6VNLDkj6f0odKmi/p8XS/U67MV9IERI9J+kAu/eA0wdDSNBGZUvqbJF2T0u+VtFeuzJRUx+OSplTafjOrqWXAnrnnewDPVpFnExHxXES8ERHrge+QDVE2s/6n4v6dFxHPpvuVwA24r5uZmdVMJXMKVGod8KWIeCcwjiyyvy8wDVgQEaPILkmYBpC2HQ/sRzah2BVpYiLIfg2cysZJyAoTjp0KrImItwOXAN9M+xoKnAMcQvbB4ZzuhhWbWUMsBEZJ2lvSELK+3lmUpxOYnFYhGAesjYgVPe20MOdA8lHgoe7ymllTlXMOKEnSNpK2KzwGjsJ93czMrGbqFhSIiBURcX96/BLwKNn1g5OA2SnbbOCY9HgSMDciXo2IJ4GlwNj0oX/7iLg7IgKYU1SmsK/rgCPSKIIPAPMjYnVErAHm0/3M5WZWZxGxDjiTbHnTR4FrI+JhSadLOj1lmwc8Qdb3vwP830J5ST8C7gbeIWmZpFPTpulpFNFi4HDgi405IjOrRDnnAEnDJS0D/h74x9TXtwd2A+6U9CDwG+DnEXFTc47EzMys9QxuRCVpWP+BwL3AboVf/yJihaTCSga7A/fkihUmIXo9PS5OL5R5Ju1rnaS1wM6UOaFRmqxoKsDIkSOrP0Az61VaLnBeUdqM3OMgW/q0VNkTukk/qZZtNLP6KeMc0EV2WUGxPwIH1Ld1ZmZm7auelw8AIGlb4MdkkxX+saesJdKih/Rqy2xM8IREZmZmZmZm1sbqGhSQtCVZQOAHEXF9Sn6ucB1wul+Z0rubhGgZm/5ykJ+caEMZSYOBHcjWMu/ThEZmZmZmZmZm7aBuQYF0bf/3gEcj4lu5TZ1AYTWAKcCNufTj04oCe5NNKPibdKnBS5LGpX1OLipT2NexwK1pCPLNwFGSdkoTDB6V0szMzMzMzMwsqeecAuOBk4Alkh5IaV8FLgSuTROFPQ0cB5AmHLoWeIRs5YIzIuKNVO6zwFXA1sAv0g2yoMPVkpaSjRA4Pu1rtaTzyWY7BjgvIlbX6TjNzMzMzMzMBqS6BQUi4k5KX9sPcEQ3ZS4ALiiRvgh4V4n0v5CCCiW2zQJmldteMzMzMzMzs3ZT94kGzczMzMzMzKx/clDAzMzMzMzMrE3Vc04Bs5Juf++hddv3oXfcXrd9m5mZmVnPJE0Avg0MAr4bERcWbd8H+E/gIOBrEXFRuWXNrD48UsDMzMzMzPpM0iDgcmAisC9wgqR9i7KtBs4CLqqirJnVgYMCZmZmZmZWC2OBpRHxRES8BswFJuUzRMTKiFgIvF5pWTOrDwcFzMzMzMysFnYHnsk9X5bSalZW0lRJiyQtWrVqVdUNNbONHBQwMzMzM7NaKLUcedSybETMjIgxETFm2LBhFTXOzEpzUMDMzMzMzGphGbBn7vkewLMNKGtmfeCggJmZmZmZ1cJCYJSkvSUNAY4HOhtQ1sz6wEEBM2sISRMkPSZpqaRpJbZL0qVp+2JJB+W2zZK0UtJDRWWGSpov6fF0v1MjjsXMKlfGOWAfSXdLelXS2ZWUNbP+ISLWAWcCNwOPAtdGxMOSTpd0OoCk4ZKWAX8P/KOkZZK2765sc47ErL0MbnYDzKz15ZYZOpJseOBCSZ0R8Ugu20RgVLodAlyZ7gGuAi4D5hTtehqwICIuTF8UpgFfrtdxmFl1yjwHFJYpO6aKsmbWT0TEPGBeUdqM3OMusksDyiprZvXnkQJm1gjlLDM0CZgTmXuAHSWNAIiIO8i+MBSbBMxOj2dT9GXCzPoNL1NmZmbWTzkoYGaNUM4yQ9UsY7RbRKwASPe7lsrk5YvMmq7uy5SB+7qZmVk1HBQws0YoZ5mhvixj1CMvX2TWdHVfpgzc183MzKrhoICZNUI5ywxVsxTRc4VLDNL9yj6208zqw8uUmZmZ9VMOCphZI5SzzFAnMDmtQjAOWFu4NKAHncCU9HgKcGMtG21mNeNlyszMzPoprz5gZnUXEeskFZYZGgTMKixRlLbPIJtt+GhgKfBn4JRCeUk/Ag4DdknLGJ0TEd8DLgSulXQq8DRwXOOOyszKVc45QNJwYBGwPbBe0heAfSPij6XKNuVAzMzMWpCDAmbWEGUsURTAGd2UPaGb9BeAI2rYTDOrEy9TZmZm1j/58gEzMzMzMzOzNuWggJmZmZmZmVmbclDAzMzMzMzMrE05KGBmZmZmZmbWpuoWFJA0S9JKSQ/l0oZKmi/p8XS/U27bVyQtlfSYpA/k0g+WtCRtu1SSUvqbJF2T0u+VtFeuzJRUx+OSCsuVmZmZmZmZmVlOPUcKXAVMKEqbBiyIiFHAgvQcSfuSrTu8XypzhaRBqcyVwFRgVLoV9nkqsCYi3g5cAnwz7WsocA5wCDAWOCcffDAzMzMzMzOzTN2WJIyIO/K/3ieTyNYaB5gN3AZ8OaXPjYhXgSclLQXGSnoK2D4i7gaQNAc4BvhFKnNu2td1wGVpFMEHgPkRsTqVmU8WSPhRrY+x1p4+79112/fIry+p277NzMysOh0dHXR1dTF8+HCmT5/e7OaYmVkbqltQoBu7RcQKgIhYIWnXlL47cE8u37KU9np6XJxeKPNM2tc6SWuBnfPpJcpsQtJUslEIjBw5svqjMjMzM8spN9C/7JGhPPfKYNat/kPZZRzoNzOzWuovEw2qRFr0kF5tmU0TI2ZGxJiIGDNs2LCyGmpmZmZmZmbWKhodFHhO0giAdL8ypS8D9szl2wN4NqXvUSJ9kzKSBgM7AKt72JeZmZlZv7LLVuvZbet17LLV+mY3xczM2lSjgwKdQGE1gCnAjbn049OKAnuTTSj4m3SpwUuSxqX5AiYXlSns61jg1ogI4GbgKEk7pQkGj0ppZmZmZv3K2fu/yIVjV3P2/i82uylmNSFpQlpNbKmkaSW2K60otlTSYkkH5bY9lVYde0DSosa23Kx91W1OAUk/IptUcBdJy8hWBLgQuFbSqcDTwHEAEfGwpGuBR4B1wBkR8Uba1WfJVjLYmmyCwV+k9O8BV6dJCVeTrV5ARKyWdD6wMOU7rzDpoJmZmZmZ1UdaPexy4Eiy0bsLJXVGxCO5bBPZuKrYIWQrjR2S2354RDzfoCabGfVdfeCEbjYd0U3+C4ALSqQvAt5VIv0vpKBCiW2zgFllN9bMzMzMzPpqLLA0Ip4AkDSXbMWwfFBgEjAnjfC9R9KOkkYUJiM3s8brLxMNmlmL6+NwwpJlJZ0raXkaZviApKMbdTxmVhkPKTZrC+WsAtZTngBukXRfWiVsM5KmSlokadGqVatq1Oz66+joYPLkyXR0dDS7KWabafSShGbWhvoynLCMspdExEUNOhQzq4KHFJu1jXJWAespz/iIeDYtWz5f0u8i4o5NMkbMBGYCjBkzpuQKY/1RV1cXy5cvb3YzzEpyUMDMGqHq4YTAXmWUNbP+bUAMKe7o6KCrq4vhw4czffr0RlVr1krKWQWs2zwRUbhfKekGsnPHHfRTT5/37rLzrls9FBjMutV/KLvcyK8vqbJlZpVxUMDMGqHUUMFDysizexllz5Q0GVgEfCki1hRXnoYgTgUYOXJklYfQeP6CYi2kL+eAFWwcUhzAf6RfCjfTXV8/+B/mlNXI7R5+nEGv/pGnn3+p7DL3/dvksvKZtYmFwKi0mthysonAP1WUp5PsvXsu2XlgbUSskLQNsEVEvJQeHwWc18C211W27KiXH7X+yUEBM2uEvgwn7KnslcD56fn5wMXAZzbL7KGGZs1W9yHFMHD7ulmriIh1ks4kWw58EDArrTJ2eto+A5gHHA0sBf4MnJKK7wbckK1CzmDghxFxU7l1lxvIq1ZfA4BedtT6MwcFzKwR+jKccEh3ZSPiuUKipO8AP6ukUfX8ANHdh4d6DjX0MEPrxwbEkOL1Q7bZ5N7MKhcR88i++OfTZuQeB3BGiXJPAAfUvYFmthmvPmBmjbBhOKGkIWTDCTuL8nQCk9MM5ONIwwl7KpvmHCj4KPBQvQ+kkXbZaj27be2hhtYSqj4HSNpG0nYAuSHFdenrfxp1FC/t91H+NOqoeuzezMysX/JIATOru74MJ+yubNr1dEmjyYYYPwX8XcMOqgE81NBaRTOHFJuZmVnPHBQws4aodjhhd2VT+kk1bqaZ1YmHFJuZmfVPvnzAzMzMzMzMrE05KGBmZmZmZmbWphwUMDMzMzMzM2tTDgqYmZmZmZmZtSkHBczMzMzMzMzalIMCZmZmZmZmZm3KSxKa1ckFnz62bvv+2vevq9u+zczMzMysfTgoYGZmNXf7ew+t6/4PveP2uu7fzMzMrF04KGBmZjbAPHrBrXXb9zu/9r667dvMzFpbR0cHXV1dDB8+nOnTpze7OVYmBwXMzMzMzMysz7q6uli+fHmzm2EVclDAzMzMzMzMulXuZYGvDB4EEq8sW1Z2GV8S2HwOCpiZmZmZmVmf7Rixyb31rj9cEuiggJmZmZmZmfXZp99Y3+wmWBUcFDAzMzOzqtRzpREPKTYza4wtmt2AepI0QdJjkpZKmtbs9pi1s976ozKXpu2LJR3UW1lJQyXNl/R4ut+pUcdjZpWpxznAzPof93WzgadlRwpIGgRcDhwJLAMWSuqMiEea2zKz9lNmf5wIjEq3Q4ArgUN6KTsNWBARF6YPD9OALzfquMysPHU8B5i1tP5wrXEl3NfNBqZWHikwFlgaEU9ExGvAXGBSk9tk1q7K6Y+TgDmRuQfYUdKIXspOAmanx7OBY+p8HGZWnXqdA8ysf3FfNxuAFC06M6SkY4EJEXFaen4ScEhEnJnLMxWYmp6+A3isyup2AZ7vQ3NdZ/+qs1n1DrQ6n4+ICeVkLLM//gy4MCLuTM8XkP3qv1d3ZSW9GBE75vaxJiI2u4TAfd11tnmdfam37H7ek3qdA0rUU4u+7v8L19mOdQ6Yvu73dNfZz+odSHV2289b9vIBQCXSNomARMRMYGafK5IWRcSYvu7HdfaPOptVb4vX2Wt/7CFPOWV75L7uOtu5zmbWm29CibSanwNq0df9f+E6XWffmlEiraZ93e/prrM/1dsqdbZyUGAZsGfu+R7As01qi1m7K6c/dpdnSA9ln5M0IiJWpKGHK2vaajOrlXqdA8ysf3FfNxuAWnlOgYXAKEl7SxoCHA90NrlNZu2qnP7YCUxOsxKPA9ZGxIpeynYCU9LjKcCN9T4QM6tKvc4BZta/uK+bDUAtO1IgItZJOhO4GRgEzIqIh+tUXZ+HMLnOflVns+pt2Tq764+STk/bZwDzgKOBpcCfgVN6Kpt2fSFwraRTgaeB4+p8KC37N3KdLV1nM+sF6noOqAf/X7hO11kl93XX2cQ6m1VvS9TZshMNmpmZmZmZmVnPWvnyATMzMzMzMzPrgYMCZmZmZmZmZm3KQYEySfqopJC0T3q+l6SHSuQbJ+leSQ9IelTSuVXU9UYq/6Ck+yX9TUp/i6Tr+nwwpet8uej5FyX9RdIOubTD0tqyxWU/JOm3qb2PSPq7vrQh/9qmOtem16Nwe381+6+y7s2Otw91FP6uD0n6L0lvrtW+S9S12f+npHMlnS3pNkljSuVt1OvdX7mf17+f59vRin29kf081ee+XgX3dff1GuzffX0AaFRfdz9vzX6e9tcWn99bdqLBOjgBuJNsJtRze8g3G/hERDwoaRDwjirqeiUiRgNI+gDwr8ChEfEscGwV+6vGCWSzwH4UuKq7TJK2JJvsYmxELJP0JmCvGrflVxHxoRrvsxnyf9cfAKcD32pqi0prlde7Gu7nJTSon0Nr/O8NlH4OrfF6V8t9vQT39Yq4rw8Mjerr7ueba5X/u4HS1/v0enukQBkkbQuMB04lO6n0ZFdgBUBEvBERj/Sx+u2BNakd+YjQXpJ+laKR+YjkCEl35CJaf1tphZLeBmwL/CPZCaYn25EFl14AiIhXI+KxSutsQ78C3i5pqKSfSFos6R5J+8OGqODVkm6V9Lik/9Pk9rY89/MeuZ9Xx/28H3Jf75H7enXc1/uhJvZ19/PW1bJ93SMFynMMcFNE/F7SakkHAau7yXsJ8Jik24CbgNkR8ZcK69ta0gPAVsAI4H0l8qwEjoyIv0gaBfwIGAN8Crg5Ii5Ikc5qhrickPb3K+AdknaNiJWlMkbEakmdwB8kLQB+BvwoItZXUW93/ja9HgUfj4j/qeH+G0rSYGAi2f/HN4DfRsQxkt4HzAFGp6z7A+OAbYDfSvp5ijjXW0u93hU4BvfzZvZzaKH/vQHQz6GFXu8KHYP7uvt6jbiv92vH0Li+7n6+uZb6vxsAfb1Pr7dHCpTnBGBuejyXHiJwEXEeWQe/hayT31RFfa9ExOiI2AeYAMyRpKI8WwLfkbQE+C9g35S+EDhF2bVQ746Il6qo/3hgbjo5XE8va79HxGnAEcBvgLOBWVXU2ZNfpdejcBuoJ5TCG8Yi4Gnge8B7gKsBIuJWYGdtvBbsxoh4JSKeB34JjK2gru7WGo1utuXTWuX1rpT7eQ8a0M+hNf73GtnPwX29Gu7rPXBfL5v7ev/XyL7ufr65Vvm/a4vP7x4p0AtJO5NF+94lKYBBZH+AK7ork/4IV0r6DrBK0s4R8UI19UfE3ZJ2AYYVbfoi8BxwAFlw5y8p/x2S3gt8ELha0r9FxJxy60vDX0YB89O5bAjwBHB5L+1cAiyRdDXwJHByuXW2kQ3XJBWUeMOAjR28uPN3d6Io5QVgp6K0oWR/m+JtQ4HnK9h3y3E/dz+voUb2c3Bfr4j7uvt6Dbmv92PN7Ovu5y2nLT6/e6RA744F5kTEWyNir4jYk+wPs0epzJI+mPtHGQW8AbxYbeXKZksdRLruJ2cHYEWKCJ6U8iDprcDKiPgOWSTroAqrPAE4Nx3rXhHxFmD3tN9S7dtW0mG5pNHAHyqss53dAZwI2cyhwPMR8ce0bZKkrdIb22FkkeSyRMTLwApJR6R9DyWLXN8J3AZ8Ovd/OoUsktnO3M/dz+upLv0c3Ner4L7uvl5P7uv9R9P6uvt5W2i5z+8eKdC7E4ALi9J+DHyV7JqdZbn0LwIfBy6R9GdgHXBiRLxRYZ2FYSoAAqZExBtFQakrgB9LOo7sH+JPKf0w4B8kvQ68DEyusO7jya6Xybshpd8LHFF0zCcAHZL+A3gltePkCuvsTfE1Mv8cEXVZ3qUJzgX+U9Ji4M9kHbzgN8DPgZHA+VH59UiTgcslXZyefyMi/kfSTGAf4MEUPV8EfCVXrpVf7+64nze/n0Pr/u+dS/36ObivV8J93X29ns7Ffb2/aHRfdz/fXCv/351Li31+V0Slo5fMrN6UXVf2ckRc1Oy2mFl9uJ+btQf3dbP2MJD7ui8fMDMzMzMzM2tTHilgZmZmZmZm1qY8UsDMzMzMzMysTTkoYGZmZmZmZtamHBQwMzMzMzMza1MOCljdSHpK0i412tfpkianxydLeks96jGzyrmvm7U+93Oz9uC+3p4GN7sBZr2RNDgiZuSSTgYeAqpZ49fM+in3dbPW535u1h7c1wcWBwWsJiT9BNgT2Ar4dkTMLNr+T8CJwDPA88B9EXGRpNHADODNwP8An4mINZJuA+4CxgOdkrYDXgaeAsYAP5D0CvDXqYrPSfowsCVwXET8Lq0VujcwAvjfwN8D44CJwHLgwxHxes1fDLMW5r5u1vrcz83ag/u6FfjyAauVz0TEwWQd/ixJOxc2SBoDfBw4EPhYylMwB/hyROwPLAHOyW3bMSIOjYiLCwkRcR2wCDgxIkZHxCtp0/MRcRBwJXB2bh9vAz4ITAK+D/wyIt4NvJLSzawy7utmrc/93Kw9uK8b4KCA1c5Zkh4E7iGLOI7KbXsPcGNEvBIRLwE/BZC0A9mJ4/aUbzbw3ly5ayqo//p0fx+wVy79FymauAQYBNyU0pcU5TOz8rivm7U+93Oz9uC+boAvH7AakHQY8H7gryPiz2no0Fb5LFXu+k8V5H013b/Bpv/XrwJExHpJr0dEpPT1+P/frCLu62atz/3crD24r1ueRwpYLewArEknlH3IrvvJuxP4sKStJG1LGvYTEWuBNZL+NuU7Cbid3r0EbFebpptZBdzXzVqf+7lZe3Bftw0cabFauAk4XdJi4DGyIUgbRMRCSZ3Ag8AfyK4pWps2TwFmSHoz8ARwShn1XZXK5CcqMbP6c183a33u52btwX3dNtDG0Rhm9SNp24h4OZ087gCmRsT9zW6XmdWW+7pZ63M/N2sP7uvtwyMFrFFmStqX7Fql2T6hmLUs93Wz1ud+btYe3NfbhEcKmJmZmZmZmbUpTzRoZmZmZmZm1qYcFDAzMzMzMzNrUw4KmJmZmZmZmbUpBwXMzMzMzMzM2pSDAmZmZmZmZmZt6v8DXkrlFjbak/0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1036.8x216 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = sns.catplot(x = \"algorithm\", y = \"val\", data = pltData, kind=\"bar\", col = \"metric\", aspect=1.2, height=3, sharey=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04cd45c1",
   "metadata": {
    "papermill": {
     "duration": 0.027155,
     "end_time": "2022-02-17T22:27:13.905697",
     "exception": false,
     "start_time": "2022-02-17T22:27:13.878542",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Statistical Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4826b740",
   "metadata": {
    "papermill": {
     "duration": 0.025572,
     "end_time": "2022-02-17T22:27:13.957066",
     "exception": false,
     "start_time": "2022-02-17T22:27:13.931494",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We will now examine these metrics in more detail and test for statistical significance.  Best practice for statistical testing for recommender systems is still a work in progress, but this method avoids most of the validity problems we know about.\n",
    "\n",
    "The Friedman $\\chi^2$ test will test the null hypothesis that there is no difference between algorithms in terms of their performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "20a9374f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:27:14.009484Z",
     "iopub.status.busy": "2022-02-17T22:27:14.009248Z",
     "iopub.status.idle": "2022-02-17T22:27:14.012726Z",
     "shell.execute_reply": "2022-02-17T22:27:14.012352Z"
    },
    "papermill": {
     "duration": 0.031404,
     "end_time": "2022-02-17T22:27:14.014168",
     "exception": false,
     "start_time": "2022-02-17T22:27:13.982764",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def metric_friedman(metric):\n",
    "    groups = [df[metric] for a, df in results.groupby('algorithm')]\n",
    "\n",
    "    stat, p = friedmanchisquare(*groups)\n",
    "    return pd.Series({'stat': stat, 'p': p}, name=metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c2ec069",
   "metadata": {
    "papermill": {
     "duration": 0.025997,
     "end_time": "2022-02-17T22:27:14.065956",
     "exception": false,
     "start_time": "2022-02-17T22:27:14.039959",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "If the test detects significance, we have at least one pair of different algorithms - but which pairs?  Unfortunately, the good tests aren't readily available in scipy and friends, so we're going to take a conservative approch and perform pairwise Wilconxon signed-rank tests with a Bonferroni adjustment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "17fced3a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:27:14.125111Z",
     "iopub.status.busy": "2022-02-17T22:27:14.124939Z",
     "iopub.status.idle": "2022-02-17T22:27:14.129784Z",
     "shell.execute_reply": "2022-02-17T22:27:14.129423Z"
    },
    "papermill": {
     "duration": 0.038502,
     "end_time": "2022-02-17T22:27:14.130867",
     "exception": false,
     "start_time": "2022-02-17T22:27:14.092365",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "pairs = pd.DataFrame.from_records(combinations(rec_algos, 2), columns=['A1', 'A2'])\n",
    "def am_wilcox(metric, p_scale = len(pairs)):\n",
    "    def compute(df):\n",
    "        vs1 = results.loc[df['A1'], metric]\n",
    "        vs2 = results.loc[df['A2'], metric]\n",
    "        diff = (vs1 - vs2).mean()\n",
    "        stat, p = wilcoxon(vs1, vs2)\n",
    "        return pd.Series({'diff': diff, 'stat': stat, 'p': p, 'adj_p': p * p_scale})\n",
    "    scores = pairs.apply(compute, axis=1)\n",
    "    return pairs.join(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab160342",
   "metadata": {
    "papermill": {
     "duration": 0.026113,
     "end_time": "2022-02-17T22:27:14.182502",
     "exception": false,
     "start_time": "2022-02-17T22:27:14.156389",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "12737d8b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:27:14.235398Z",
     "iopub.status.busy": "2022-02-17T22:27:14.235211Z",
     "iopub.status.idle": "2022-02-17T22:27:17.879457Z",
     "shell.execute_reply": "2022-02-17T22:27:17.879087Z"
    },
    "papermill": {
     "duration": 3.672248,
     "end_time": "2022-02-17T22:27:17.880893",
     "exception": false,
     "start_time": "2022-02-17T22:27:14.208645",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "stat    262913.826905\n",
       "p            0.000000\n",
       "Name: precision, dtype: float64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metric_friedman('precision')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96cb5751",
   "metadata": {
    "papermill": {
     "duration": 0.028514,
     "end_time": "2022-02-17T22:27:17.935252",
     "exception": false,
     "start_time": "2022-02-17T22:27:17.906738",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "This indicates a statistically significant difference exists - but where is it?\n",
    "\n",
    "To answer that question, we need a post-hoc test. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7760c854",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:27:17.989844Z",
     "iopub.status.busy": "2022-02-17T22:27:17.989625Z",
     "iopub.status.idle": "2022-02-17T22:27:18.253178Z",
     "shell.execute_reply": "2022-02-17T22:27:18.252808Z"
    },
    "papermill": {
     "duration": 0.292243,
     "end_time": "2022-02-17T22:27:18.254325",
     "exception": false,
     "start_time": "2022-02-17T22:27:17.962082",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A1</th>\n",
       "      <th>A2</th>\n",
       "      <th>diff</th>\n",
       "      <th>stat</th>\n",
       "      <th>p</th>\n",
       "      <th>adj_p</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Pop</td>\n",
       "      <td>UUE</td>\n",
       "      <td>0.015545</td>\n",
       "      <td>38610808.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Pop</td>\n",
       "      <td>IIE</td>\n",
       "      <td>0.015295</td>\n",
       "      <td>41820584.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Pop</td>\n",
       "      <td>ALS</td>\n",
       "      <td>0.012906</td>\n",
       "      <td>90860867.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Pop</td>\n",
       "      <td>Bias</td>\n",
       "      <td>0.012754</td>\n",
       "      <td>94484783.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Bias</td>\n",
       "      <td>UUE</td>\n",
       "      <td>0.002791</td>\n",
       "      <td>107733040.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Bias</td>\n",
       "      <td>IIE</td>\n",
       "      <td>0.002542</td>\n",
       "      <td>39920037.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>IIE</td>\n",
       "      <td>UUE</td>\n",
       "      <td>0.000250</td>\n",
       "      <td>99319368.0</td>\n",
       "      <td>2.147777e-19</td>\n",
       "      <td>4.510332e-18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Bias</td>\n",
       "      <td>ALS</td>\n",
       "      <td>0.000153</td>\n",
       "      <td>56174361.5</td>\n",
       "      <td>1.698976e-14</td>\n",
       "      <td>3.567850e-13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>IIE</td>\n",
       "      <td>ALS</td>\n",
       "      <td>-0.002389</td>\n",
       "      <td>25079696.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>UUE</td>\n",
       "      <td>ALS</td>\n",
       "      <td>-0.002639</td>\n",
       "      <td>107377077.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Pop</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.006281</td>\n",
       "      <td>225112322.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>III</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.007736</td>\n",
       "      <td>113325554.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pop</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.014017</td>\n",
       "      <td>154813327.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Bias</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.019034</td>\n",
       "      <td>33170232.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>ALS</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.019187</td>\n",
       "      <td>29097760.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>IIE</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.021576</td>\n",
       "      <td>14585484.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>UUE</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.021826</td>\n",
       "      <td>16678470.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Bias</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.026771</td>\n",
       "      <td>10806690.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>ALS</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.026923</td>\n",
       "      <td>9619085.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>IIE</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.029312</td>\n",
       "      <td>4564054.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>UUE</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.029562</td>\n",
       "      <td>4197542.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      A1    A2      diff         stat             p         adj_p\n",
       "2    Pop   UUE  0.015545   38610808.0  0.000000e+00  0.000000e+00\n",
       "1    Pop   IIE  0.015295   41820584.5  0.000000e+00  0.000000e+00\n",
       "3    Pop   ALS  0.012906   90860867.5  0.000000e+00  0.000000e+00\n",
       "0    Pop  Bias  0.012754   94484783.5  0.000000e+00  0.000000e+00\n",
       "7   Bias   UUE  0.002791  107733040.0  0.000000e+00  0.000000e+00\n",
       "6   Bias   IIE  0.002542   39920037.0  0.000000e+00  0.000000e+00\n",
       "11   IIE   UUE  0.000250   99319368.0  2.147777e-19  4.510332e-18\n",
       "8   Bias   ALS  0.000153   56174361.5  1.698976e-14  3.567850e-13\n",
       "12   IIE   ALS -0.002389   25079696.5  0.000000e+00  0.000000e+00\n",
       "15   UUE   ALS -0.002639  107377077.5  0.000000e+00  0.000000e+00\n",
       "4    Pop   III -0.006281  225112322.5  0.000000e+00  0.000000e+00\n",
       "20   III  IALS -0.007736  113325554.0  0.000000e+00  0.000000e+00\n",
       "5    Pop  IALS -0.014017  154813327.5  0.000000e+00  0.000000e+00\n",
       "9   Bias   III -0.019034   33170232.0  0.000000e+00  0.000000e+00\n",
       "18   ALS   III -0.019187   29097760.0  0.000000e+00  0.000000e+00\n",
       "13   IIE   III -0.021576   14585484.5  0.000000e+00  0.000000e+00\n",
       "16   UUE   III -0.021826   16678470.5  0.000000e+00  0.000000e+00\n",
       "10  Bias  IALS -0.026771   10806690.0  0.000000e+00  0.000000e+00\n",
       "19   ALS  IALS -0.026923    9619085.5  0.000000e+00  0.000000e+00\n",
       "14   IIE  IALS -0.029312    4564054.5  0.000000e+00  0.000000e+00\n",
       "17   UUE  IALS -0.029562    4197542.5  0.000000e+00  0.000000e+00"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "am_wilcox('precision').sort_values('diff', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd9e3de3",
   "metadata": {
    "papermill": {
     "duration": 0.026593,
     "end_time": "2022-02-17T22:27:18.307354",
     "exception": false,
     "start_time": "2022-02-17T22:27:18.280761",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can consider pairs with an adjusted $p$ value less than 0.05 (5.0e-2) to be statistically significant.  However, we have so many data points, the substance of significance is questionable.  Pay attention to the 'diff' column, which is the difference between A1 and A2 (how much A1 outperforms A2)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5532c308",
   "metadata": {
    "papermill": {
     "duration": 0.027361,
     "end_time": "2022-02-17T22:27:18.361496",
     "exception": false,
     "start_time": "2022-02-17T22:27:18.334135",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### nDCG"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce5de432",
   "metadata": {
    "papermill": {
     "duration": 0.026579,
     "end_time": "2022-02-17T22:27:18.414694",
     "exception": false,
     "start_time": "2022-02-17T22:27:18.388115",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now let's do the same for nDCG:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "6f61160c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:27:18.470386Z",
     "iopub.status.busy": "2022-02-17T22:27:18.470173Z",
     "iopub.status.idle": "2022-02-17T22:27:22.151064Z",
     "shell.execute_reply": "2022-02-17T22:27:22.150677Z"
    },
    "papermill": {
     "duration": 3.709857,
     "end_time": "2022-02-17T22:27:22.152335",
     "exception": false,
     "start_time": "2022-02-17T22:27:18.442478",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "stat    234436.773049\n",
       "p            0.000000\n",
       "Name: ndcg, dtype: float64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metric_friedman('ndcg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "fda801cf",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:27:22.215214Z",
     "iopub.status.busy": "2022-02-17T22:27:22.215003Z",
     "iopub.status.idle": "2022-02-17T22:27:22.586881Z",
     "shell.execute_reply": "2022-02-17T22:27:22.586503Z"
    },
    "papermill": {
     "duration": 0.403703,
     "end_time": "2022-02-17T22:27:22.588067",
     "exception": false,
     "start_time": "2022-02-17T22:27:22.184364",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A1</th>\n",
       "      <th>A2</th>\n",
       "      <th>diff</th>\n",
       "      <th>stat</th>\n",
       "      <th>p</th>\n",
       "      <th>adj_p</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Pop</td>\n",
       "      <td>UUE</td>\n",
       "      <td>0.140720</td>\n",
       "      <td>46637395.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Pop</td>\n",
       "      <td>IIE</td>\n",
       "      <td>0.137795</td>\n",
       "      <td>56508944.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Pop</td>\n",
       "      <td>ALS</td>\n",
       "      <td>0.109444</td>\n",
       "      <td>209154898.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Pop</td>\n",
       "      <td>Bias</td>\n",
       "      <td>0.104823</td>\n",
       "      <td>243592982.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Bias</td>\n",
       "      <td>UUE</td>\n",
       "      <td>0.035897</td>\n",
       "      <td>112625106.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Bias</td>\n",
       "      <td>IIE</td>\n",
       "      <td>0.032972</td>\n",
       "      <td>57313793.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Bias</td>\n",
       "      <td>ALS</td>\n",
       "      <td>0.004621</td>\n",
       "      <td>198413255.0</td>\n",
       "      <td>5.746133e-25</td>\n",
       "      <td>1.206688e-23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>IIE</td>\n",
       "      <td>UUE</td>\n",
       "      <td>0.002925</td>\n",
       "      <td>110026263.5</td>\n",
       "      <td>1.459984e-44</td>\n",
       "      <td>3.065967e-43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>IIE</td>\n",
       "      <td>ALS</td>\n",
       "      <td>-0.028351</td>\n",
       "      <td>36879491.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>III</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.030622</td>\n",
       "      <td>918473943.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>UUE</td>\n",
       "      <td>ALS</td>\n",
       "      <td>-0.031276</td>\n",
       "      <td>104487939.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Pop</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.092766</td>\n",
       "      <td>483168134.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pop</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.123388</td>\n",
       "      <td>463048262.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Bias</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.197589</td>\n",
       "      <td>125010440.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>ALS</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.202210</td>\n",
       "      <td>90415668.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Bias</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.228211</td>\n",
       "      <td>85738215.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>IIE</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.230562</td>\n",
       "      <td>20403854.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>ALS</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.232832</td>\n",
       "      <td>64838213.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>UUE</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.233486</td>\n",
       "      <td>18197971.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>IIE</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.261184</td>\n",
       "      <td>11452983.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>UUE</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.264108</td>\n",
       "      <td>5712849.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      A1    A2      diff         stat             p         adj_p\n",
       "2    Pop   UUE  0.140720   46637395.0  0.000000e+00  0.000000e+00\n",
       "1    Pop   IIE  0.137795   56508944.0  0.000000e+00  0.000000e+00\n",
       "3    Pop   ALS  0.109444  209154898.0  0.000000e+00  0.000000e+00\n",
       "0    Pop  Bias  0.104823  243592982.0  0.000000e+00  0.000000e+00\n",
       "7   Bias   UUE  0.035897  112625106.5  0.000000e+00  0.000000e+00\n",
       "6   Bias   IIE  0.032972   57313793.0  0.000000e+00  0.000000e+00\n",
       "8   Bias   ALS  0.004621  198413255.0  5.746133e-25  1.206688e-23\n",
       "11   IIE   UUE  0.002925  110026263.5  1.459984e-44  3.065967e-43\n",
       "12   IIE   ALS -0.028351   36879491.5  0.000000e+00  0.000000e+00\n",
       "20   III  IALS -0.030622  918473943.5  0.000000e+00  0.000000e+00\n",
       "15   UUE   ALS -0.031276  104487939.0  0.000000e+00  0.000000e+00\n",
       "4    Pop   III -0.092766  483168134.5  0.000000e+00  0.000000e+00\n",
       "5    Pop  IALS -0.123388  463048262.5  0.000000e+00  0.000000e+00\n",
       "9   Bias   III -0.197589  125010440.5  0.000000e+00  0.000000e+00\n",
       "18   ALS   III -0.202210   90415668.5  0.000000e+00  0.000000e+00\n",
       "10  Bias  IALS -0.228211   85738215.0  0.000000e+00  0.000000e+00\n",
       "13   IIE   III -0.230562   20403854.5  0.000000e+00  0.000000e+00\n",
       "19   ALS  IALS -0.232832   64838213.0  0.000000e+00  0.000000e+00\n",
       "16   UUE   III -0.233486   18197971.5  0.000000e+00  0.000000e+00\n",
       "14   IIE  IALS -0.261184   11452983.0  0.000000e+00  0.000000e+00\n",
       "17   UUE  IALS -0.264108    5712849.0  0.000000e+00  0.000000e+00"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "am_wilcox('ndcg').sort_values('diff', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad2a3bfe",
   "metadata": {
    "papermill": {
     "duration": 0.029322,
     "end_time": "2022-02-17T22:27:22.645351",
     "exception": false,
     "start_time": "2022-02-17T22:27:22.616029",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Reciprocal Rank\n",
    "\n",
    "We'll do this also for reciprocal rank (what is aggregated to compute MRR):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "abb041a9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:27:22.701505Z",
     "iopub.status.busy": "2022-02-17T22:27:22.701294Z",
     "iopub.status.idle": "2022-02-17T22:27:26.331862Z",
     "shell.execute_reply": "2022-02-17T22:27:26.331488Z"
    },
    "papermill": {
     "duration": 3.660117,
     "end_time": "2022-02-17T22:27:26.333180",
     "exception": false,
     "start_time": "2022-02-17T22:27:22.673063",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "stat    214890.780089\n",
       "p            0.000000\n",
       "Name: recip_rank, dtype: float64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metric_friedman('recip_rank')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "c3c82ce8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:27:26.389782Z",
     "iopub.status.busy": "2022-02-17T22:27:26.389564Z",
     "iopub.status.idle": "2022-02-17T22:27:26.703089Z",
     "shell.execute_reply": "2022-02-17T22:27:26.702717Z"
    },
    "papermill": {
     "duration": 0.343577,
     "end_time": "2022-02-17T22:27:26.704498",
     "exception": false,
     "start_time": "2022-02-17T22:27:26.360921",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A1</th>\n",
       "      <th>A2</th>\n",
       "      <th>diff</th>\n",
       "      <th>stat</th>\n",
       "      <th>p</th>\n",
       "      <th>adj_p</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Pop</td>\n",
       "      <td>UUE</td>\n",
       "      <td>0.170884</td>\n",
       "      <td>44889423.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Pop</td>\n",
       "      <td>IIE</td>\n",
       "      <td>0.167326</td>\n",
       "      <td>67098501.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Pop</td>\n",
       "      <td>ALS</td>\n",
       "      <td>0.124807</td>\n",
       "      <td>291609550.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Pop</td>\n",
       "      <td>Bias</td>\n",
       "      <td>0.105475</td>\n",
       "      <td>341193646.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Bias</td>\n",
       "      <td>UUE</td>\n",
       "      <td>0.065409</td>\n",
       "      <td>103127570.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Bias</td>\n",
       "      <td>IIE</td>\n",
       "      <td>0.061851</td>\n",
       "      <td>59219212.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>III</td>\n",
       "      <td>IALS</td>\n",
       "      <td>0.037939</td>\n",
       "      <td>916556715.5</td>\n",
       "      <td>2.626652e-42</td>\n",
       "      <td>5.515969e-41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Bias</td>\n",
       "      <td>ALS</td>\n",
       "      <td>0.019332</td>\n",
       "      <td>191162112.5</td>\n",
       "      <td>1.350944e-33</td>\n",
       "      <td>2.836981e-32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>IIE</td>\n",
       "      <td>UUE</td>\n",
       "      <td>0.003559</td>\n",
       "      <td>106072790.0</td>\n",
       "      <td>3.784110e-72</td>\n",
       "      <td>7.946630e-71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>IIE</td>\n",
       "      <td>ALS</td>\n",
       "      <td>-0.042518</td>\n",
       "      <td>36612230.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>UUE</td>\n",
       "      <td>ALS</td>\n",
       "      <td>-0.046077</td>\n",
       "      <td>92187285.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pop</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.099837</td>\n",
       "      <td>683352574.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Pop</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.137776</td>\n",
       "      <td>520122602.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Bias</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.205312</td>\n",
       "      <td>264661805.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>ALS</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.224644</td>\n",
       "      <td>206154836.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Bias</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.243251</td>\n",
       "      <td>240821983.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>ALS</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.262583</td>\n",
       "      <td>171129102.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>IIE</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.267163</td>\n",
       "      <td>33756363.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>UUE</td>\n",
       "      <td>IALS</td>\n",
       "      <td>-0.270721</td>\n",
       "      <td>10307060.5</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>IIE</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.305101</td>\n",
       "      <td>32266827.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>UUE</td>\n",
       "      <td>III</td>\n",
       "      <td>-0.308660</td>\n",
       "      <td>18752349.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      A1    A2      diff         stat             p         adj_p\n",
       "2    Pop   UUE  0.170884   44889423.0  0.000000e+00  0.000000e+00\n",
       "1    Pop   IIE  0.167326   67098501.0  0.000000e+00  0.000000e+00\n",
       "3    Pop   ALS  0.124807  291609550.5  0.000000e+00  0.000000e+00\n",
       "0    Pop  Bias  0.105475  341193646.5  0.000000e+00  0.000000e+00\n",
       "7   Bias   UUE  0.065409  103127570.5  0.000000e+00  0.000000e+00\n",
       "6   Bias   IIE  0.061851   59219212.0  0.000000e+00  0.000000e+00\n",
       "20   III  IALS  0.037939  916556715.5  2.626652e-42  5.515969e-41\n",
       "8   Bias   ALS  0.019332  191162112.5  1.350944e-33  2.836981e-32\n",
       "11   IIE   UUE  0.003559  106072790.0  3.784110e-72  7.946630e-71\n",
       "12   IIE   ALS -0.042518   36612230.5  0.000000e+00  0.000000e+00\n",
       "15   UUE   ALS -0.046077   92187285.5  0.000000e+00  0.000000e+00\n",
       "5    Pop  IALS -0.099837  683352574.0  0.000000e+00  0.000000e+00\n",
       "4    Pop   III -0.137776  520122602.5  0.000000e+00  0.000000e+00\n",
       "10  Bias  IALS -0.205312  264661805.0  0.000000e+00  0.000000e+00\n",
       "19   ALS  IALS -0.224644  206154836.0  0.000000e+00  0.000000e+00\n",
       "9   Bias   III -0.243251  240821983.0  0.000000e+00  0.000000e+00\n",
       "18   ALS   III -0.262583  171129102.0  0.000000e+00  0.000000e+00\n",
       "14   IIE  IALS -0.267163   33756363.0  0.000000e+00  0.000000e+00\n",
       "17   UUE  IALS -0.270721   10307060.5  0.000000e+00  0.000000e+00\n",
       "13   IIE   III -0.305101   32266827.0  0.000000e+00  0.000000e+00\n",
       "16   UUE   III -0.308660   18752349.0  0.000000e+00  0.000000e+00"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "am_wilcox('recip_rank').sort_values('diff', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79ef07c6",
   "metadata": {
    "papermill": {
     "duration": 0.029673,
     "end_time": "2022-02-17T22:27:26.762300",
     "exception": false,
     "start_time": "2022-02-17T22:27:26.732627",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Prediction RMSE\n",
    "\n",
    "We will also look at the prediction RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1375edd9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:27:26.821867Z",
     "iopub.status.busy": "2022-02-17T22:27:26.821640Z",
     "iopub.status.idle": "2022-02-17T22:30:43.342216Z",
     "shell.execute_reply": "2022-02-17T22:30:43.341422Z"
    },
    "papermill": {
     "duration": 196.551928,
     "end_time": "2022-02-17T22:30:43.344130",
     "exception": false,
     "start_time": "2022-02-17T22:27:26.792202",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "user_rmse = preds.groupby(['algorithm', 'user']).apply(lambda df: rmse(df['prediction'], df['rating']))\n",
    "user_rmse = user_rmse.reset_index(name='RMSE')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "fd3dddfb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-17T22:30:43.403591Z",
     "iopub.status.busy": "2022-02-17T22:30:43.403345Z",
     "iopub.status.idle": "2022-02-17T22:30:45.938406Z",
     "shell.execute_reply": "2022-02-17T22:30:45.938042Z"
    },
    "papermill": {
     "duration": 2.565896,
     "end_time": "2022-02-17T22:30:45.939618",
     "exception": false,
     "start_time": "2022-02-17T22:30:43.373722",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x7fffccd87070>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFgCAYAAACFYaNMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAXaUlEQVR4nO3de7SddX3n8ffHJBQUNWNJV5BwG1dGlzcuZiEMozJeZgLiZKrYgalE6XSluLyPNq1O66Vr2ulQ284glkysLom6pE5xmNQBrGtGBDoDGpBwkeqKVocEMgSQYCQqge/8sZ/YzeacnKDn2b99Tt6vtfY6z+W3n/3d2Tmf/PLbz/N7UlVIksbvSa0LkKQDlQEsSY0YwJLUiAEsSY0YwJLUyMLWBTxRK1eurKuuuqp1GZL0RGSqjXOuB3zvvfe2LkGSZsWcC2BJmi8MYElqxACWpEYMYElqxACWpEYMYElqxACWpEYMYElqxACWpEYMYElqxACWpEYMYElqZM7NhiaprbVr17J9+3aWLl3KBRdc0LqcOa33HnCSBUm+nuQLU+xLkguTbElyS5IT+65H0s9n+/btbNu2je3bt7cuZc4bRw/4HcAdwNOm2Hc6sLx7vBi4uPspaRZd9O6/mrVjPXDvD3/6c7aO+9Y/fs2sHGeu6bUHnGQZ8Grgz6dpsgrYUAPXA4uTHN5nTZI0KfoegvhPwFrg0Wn2HwHcObS+tdv2GEnWJNmUZNOOHTtmvUhJ++8pBz2Np/zCYp5y0FT/qdUT0dsQRJIzgXuq6sYkp03XbIpt9bgNVeuB9QArVqx43H5J43Pqs17buoR5o88e8KnAv0jyXeBS4OVJPj3SZitw5ND6MuCuHmuSpInRWwBX1XurallVHQOcDfyvqnrDSLONwOrubIiTgZ1VdXdfNUnSJBn7ecBJzgeoqnXAFcAZwBbgIeC8cdcjSa2MJYCr6mrg6m553dD2At4yjhokadJ4KbIkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNdJbACc5OMlXk2xOcnuSD03R5rQkO5Pc3D3e31c9kjRpFvZ47B8DL6+qXUkWAdclubKqrh9pd21VndljHZI0kXoL4KoqYFe3uqh7VF+vJ0lzTa9jwEkWJLkZuAf4UlXdMEWzU7phiiuTPG+a46xJsinJph07dvRZsiSNTa8BXFWPVNXxwDLgpCTPH2lyE3B0VR0HfAS4fJrjrK+qFVW1YsmSJX2WLEljM5azIKrqAeBqYOXI9gerale3fAWwKMlh46hJklrr8yyIJUkWd8uHAK8E/nakzdIk6ZZP6uq5r6+aJGmS9HkWxOHAJUkWMAjWz1XVF5KcD1BV64CzgDcn2QPsBs7uvryTpHmvz7MgbgFOmGL7uqHli4CL+qpBkiaZV8JJUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiO9BXCSg5N8NcnmJLcn+dAUbZLkwiRbktyS5MS+6pGkSbOwx2P/GHh5Ve1Ksgi4LsmVVXX9UJvTgeXd48XAxd1PSZr3eusB18CubnVR96iRZquADV3b64HFSQ7vqyZJmiS9jgEnWZDkZuAe4EtVdcNIkyOAO4fWt3bbRo+zJsmmJJt27NjRW72SNE69BnBVPVJVxwPLgJOSPH+kSaZ62hTHWV9VK6pqxZIlS3qoVJNs7dq1rF69mrVr17YuRZpVfY4B/1RVPZDkamAlcNvQrq3AkUPry4C7xlGT5o7t27ezbdu21mVIs663AE6yBHi4C99DgFcC/3Gk2UbgrUkuZfDl286quruvmjQ+p37k1Fk71kEPHMSTeBJ3PnDnrB33b972N7NyHOnn0WcP+HDgkiQLGAx1fK6qvpDkfICqWgdcAZwBbAEeAs7rsR7NUfXk4lEepZ78uNEpaU7rLYCr6hbghCm2rxtaLuAtfdWg+eHhUx9uXYLUC6+Ek6RGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJaqS3AE5yZJIvJ7kjye1J3jFFm9OS7Exyc/d4f1/1SNKkWdjjsfcA766qm5I8FbgxyZeq6hsj7a6tqjN7rEOSJlJvPeCquruqbuqWfwDcARzR1+tJ0lwzljHgJMcAJwA3TLH7lCSbk1yZ5HnTPH9Nkk1JNu3YsaPPUiVpbHoP4CSHApcB76yqB0d23wQcXVXHAR8BLp/qGFW1vqpWVNWKJUuW9FqvJI1LrwGcZBGD8P1MVX1+dH9VPVhVu7rlK4BFSQ7rsyZJmhR9ngUR4OPAHVX1J9O0Wdq1I8lJXT339VWTJE2SPs+COBU4F7g1yc3dtvcBRwFU1TrgLODNSfYAu4Gzq6p6rEmSJkZvAVxV1wGZoc1FwEV91SBJk8wr4SSpEQNYkhoxgCWpEQNYkhoxgCWpEQNYkhoxgCWpEQNYkhoxgCWpEQNYkhrZZwAnefnQ8rEj+17bV1GSdCCYqQf84aHly0b2/c4s1yJJB5SZAjjTLE+1Lkl6AmYK4Jpmeap1SdITMNN0lP8wyUYGvd29y3Trx07/NEnSTGYK4FVDyx8e2Te6Lkl6AvYZwFX1leH17h5vzwe2VdU9fRYmSfPdTKehrdt7q/gkTwc2AxuAryc5Zwz1SdK8NdOXcC+pqtu75fOAb1XVC4AXAWt7rUyS5rmZAvgnQ8uvAi4HqKrtfRUkSQeKmQL4gSRnJjmBwV2OrwJIshA4pO/iJGk+m+ksiN8ALgSWAu8c6vm+AvgffRYmSfPdTGdBfAtYOcX2LwJf7KsoSToQ7DOAk1y4r/1V9fbZLUeSDhwzDUGcD9wGfA64C+d/kKRZM1MAHw68HvhXwB7gL4DLqur7fRcmSfPdPs+CqKr7qmpdVf1T4E3AYuD2JOeOoTZJmtf2644YSU4E3gm8AbgSuHE/nnNkki8nuSPJ7UneMUWbJLkwyZYkt3SvI0kHhJm+hPsQcCZwB3Ap8N6q2rOfx94DvLuqbkryVODGJF+qqm8MtTkdWN49Xgxc3P2UpHlvpjHg3wW+AxzXPf4gCQy+jKuqeuF0T6yqu4G7u+UfJLkDOAIYDuBVwIaqKuD6JIuTHN49V5LmtZkCeFbm/E1yDHACcMPIriOAO4fWt3bbHhPASdYAawCOOuqo2ShJkpqb6UKM7021PckC4Gxgyv0jbQ9lcD+5d1bVg6O7p3rZKepYD6wHWLFihXfikDQvzDQd5dOSvDfJRUn+Wfel2dsYDEv8ykwH7+YPvgz4TFV9foomW4Ejh9aXMTjfWJLmvZnOgvgU8GzgVuDXgb8GzgJWVdWqfT0xg8HijwN3VNWfTNNsI7C6C/aTgZ2O/0o6UMx4T7hu/l+S/DlwL3BUVf1gP459KnAucGuSm7tt7wOOAqiqdcAVwBnAFuAhBnMOS9IBYaYAfnjvQlU9kuTv9jN8qarrmOHS5e7sh7fsz/Ekab6ZKYCPS7L3i7MAh3Tre09De1qv1UnSPDbTWRALxlWIJB1o9utSZEnS7DOAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGuktgJN8Isk9SW6bZv9pSXYmubl7vL+vWiRpEi3s8difBC4CNuyjzbVVdWaPNUjSxOqtB1xV1wD393V8SZrrWo8Bn5Jkc5IrkzxvukZJ1iTZlGTTjh07xlmfJPWmZQDfBBxdVccBHwEun65hVa2vqhVVtWLJkiXjqk+SetUsgKvqwara1S1fASxKclireiRp3JoFcJKlSdItn9TVcl+reiRp3Ho7CyLJZ4HTgMOSbAU+ACwCqKp1wFnAm5PsAXYDZ1dV9VWPJE2a3gK4qs6ZYf9FDE5Tk6QDUuuzICTpgGUAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNWIAS1IjBrAkNdJbACf5RJJ7ktw2zf4kuTDJliS3JDmxr1okaRL12QP+JLByH/tPB5Z3jzXAxT3WIkkTp7cArqprgPv30WQVsKEGrgcWJzm8r3okadK0HAM+ArhzaH1rt+1xkqxJsinJph07doylOEnqW8sAzhTbaqqGVbW+qlZU1YolS5b0XJYkjUfLAN4KHDm0vgy4q1EtkjR2LQN4I7C6OxviZGBnVd3dsB5JGquFfR04yWeB04DDkmwFPgAsAqiqdcAVwBnAFuAh4Ly+apGkSdRbAFfVOTPsL+Atfb2+JE06r4STpEYMYElqxACWpEYMYElqxACWpEZ6OwtC7a1du5bt27ezdOlSLrjggtblSBphAM9j27dvZ9u2ba3LkDQNA3iC/N/fe8GsHm/P/c8AFrLn/u/N2rGPev+ts3IcSQbwvHbYwY8Ce7qfkiaNATyPveeFD7QuQUMck//5/f4bzmpdwoz+3af/cr/bzosAftFvbmhdwj7d+EerW5egCeCYvEbNiwCW+vKVl75s1o61e+ECSNi9deusHfdl13xlVo6jNgxgaUwWVz3mp2QAS2Pyhkf8MlSP5ZVwktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktRIrwGcZGWSbybZkuS3p9h/WpKdSW7uHu/vsx5JmiS9TUeZZAHwUeBVwFbga0k2VtU3RppeW1Vn9lWHJE2qPnvAJwFbquo7VfUT4FJgVY+vJ0lzSp8BfARw59D61m7bqFOSbE5yZZLn9ViPJE2UPu+IkSm2jd6L5Sbg6KraleQM4HJg+eMOlKwB1gAcddRRs1ymJLXRZw94K3Dk0Poy4K7hBlX1YFXt6pavABYlOWz0QFW1vqpWVNWKJUuW9FiyJI1PnwH8NWB5kmOTHAScDWwcbpBkaZJ0yyd19dzXY02SNDF6G4Koqj1J3gp8EVgAfKKqbk9yfrd/HXAW8OYke4DdwNlV3jJW0oGh17sid8MKV4xsWze0fBFwUZ81SNKk8ko4SWrEAJakRgxgSWrEAJakRgxgSWrEAJakRgxgSWrEAJakRgxgSWrEAJakRgxgSWrEAJakRgxgSWrEAJakRgxgSWrEAJakRgxgSWrEAJakRgxgSWrEAJakRgxgSWrEAJakRgxgSWrEAJakRgxgSWrEAJakRgxgSWrEAJakRnoN4CQrk3wzyZYkvz3F/iS5sNt/S5IT+6xHkiZJbwGcZAHwUeB04LnAOUmeO9LsdGB591gDXNxXPZI0afrsAZ8EbKmq71TVT4BLgVUjbVYBG2rgemBxksN7rEmSJkaqqp8DJ2cBK6vq17v1c4EXV9Vbh9p8AfjDqrquW/+fwG9V1aaRY61h0EMGeDbwzV6K/nuHAff2/BrjMl/ei+9jsvg+nph7q2rl6MaFPb5gptg2mvb704aqWg+sn42i9keSTVW1Ylyv16f58l58H5PF9zE7+hyC2AocObS+DLjrZ2gjSfNSnwH8NWB5kmOTHAScDWwcabMRWN2dDXEysLOq7u6xJkmaGL0NQVTVniRvBb4ILAA+UVW3Jzm/278OuAI4A9gCPASc11c9T9DYhjvGYL68F9/HZPF9zILevoSTJO2bV8JJUiMGsCQ1csAGcJJfTlJJntOtH5PktinanZzkhiQ3J7kjyQfHXuwUkjzS1bQ5yU1J/nG3/ZlJ/rJ1ffsrya6R9Xcl+VGSpw9tO607Z3z0uWcm+Xr3Z/CNJL8xjpr31973Nvx3q3svO7vPbu/jlW0r3T/7eD+P+2wmwVS/00k+mOQ9Sa5OsmKqtuP8jPo8D3jSnQNcx+DsjA/uo90lwK9U1ebu8upnj6G2/bG7qo4HSPLPgf8AvKyq7gLOalnYz+kcBmfQ/DLwyekaJVnE4AuUk6pqa5JfAI4ZR4Gz4NqqOrN1EdqnsXxGB2QPOMmhwKnAv2EQwPvyS8DdAFX1SFV9o+fyfhZPA74Pj/uX/Jgk13Y95OFe8uFJrun+Zb8tyUsa1v5TSZ4FHAr8DoMg3penMuhA3AdQVT+uqr6vkJRm1YHaA/6XwFVV9a0k93ezsN0/Tds/Bb6Z5GrgKuCSqvrReMrcp0OS3AwcDBwOvHyKNvcAr6qqHyVZDnwWWAH8a+CLVfX7Xa/+yWOqeSbnMKjxWuDZSX6pqu6ZqmFV3Z9kI/C97hL2LwCfrapHx1fuz+wl3We31+uq6tutitGUxvIZHZA9YAa/6Jd2y5eyj95WVf0eg9D6awbBdVXv1e2f3VV1fFU9B1gJbEgyemn3IuBjSW4F/iuDWelg8F/887rx7BdU1Q/GVfQMzgYu7UL088Dr99W4m2fkFcBXgfcAn+i9wtlxbffZ7X0Yvv2Y7hzbmmbf8LaxfEYHXA84yS8y6C0+P0kxuEikgD+b7jndH/7FST4G7Ejyi1V131gK3g9V9X+SHAYsGdn1LuD/Accx+Mf2R137a5K8FHg18Kkkf1RVG8ZZ86gkL2QwLemXun9HDgK+w2BK02lV1a3ArUk+Bfwd8KZ+K9Ucch/wD0a2PYPB35PRfc+gweRCB2IP+CwGU2AeXVXHVNWRDD6QZVM1TvLqoZ7lcuAR4IGxVLqfujM5FtCNhw55OnB316M8t2tDkqOBe6rqY8DHgUmYCP8c4IPdZ3JMVT0TOKKr9XGSHJrktKFNxwPf671KzRlVtQu4O8krAJI8g8H/Fq8DrgbeMPS7/Ubgy+Ou8YDrATP4Rf/DkW2XAe9jMO64dWj7u4DXAX+a5CFgD/CrVfXIWCrdt0OGxqgCvLGqHhkZhfgz4LIkr2fwl+uH3fbTgN9M8jCwC1g9lor37WwGE/QP+2/d9huAV4x8NucAa5P8F2A3g/f2pjHUORtGxxf/fVXNmVMH55jVwEeT/HG3/qGq+naS9cBzgM3d/4Q3Ae8det5YPiMvRZakRg7EIQhJmggGsCQ1YgBLUiMGsCQ1YgBLUiMGsOa8JN/tLkSZjWOdn2R1t/ymJM/s43UkODDPA5amlGRhd6usvd4E3IY3ilVPDGDNKUkuZ3An7YOB/1xV60f2/y7wq8CdDC4tvbGqPpzkeGAdg4mHvg38WlV9v5tk6X8zmB1vY5KnMrg45bsM5gD5TJLdwCndS7wtyWsYzLPx+qr6225OjWMZTIr0j4B/C5zM4MKSbcBrqurhWf/D0JznEITmml+rqhcxCMe3d3N7ANBNsP064ATgtV2bvTYAv1VVLwRuBT4wtG9xVb2sqvZeLUV31dMmBlc+Hl9Vu7td91bVicDFDCYA2utZDObWWAV8GvhyVb2AwVV6r56F9615yADWXPP2JJuB6xn0hJcP7fsnwH+vqt3dDG9/BZDB3TUWV9VXunaXAC8det5fPIHX/3z380YeOwH8lV0v91YGc27snTXvVubORPEaM4cgNGd0k++8Ejilqh7qhg8OHm7yMx76hzM3+akfdz8f4bG/Pz8GqKpHkzxcf3+N/6P4e6Zp2APWXPJ04Ptd+D6HwTjrsOuA1yQ5uLvryasBqmon8P2hO3+cC3yFmf2AwZ03pF74L7PmkquA85PcAnyTwTDET1XV17q7ZGxmMDXlJmBnt/uNwLokT2Ywz/B5+/F6n+yeM/wlnDRrnA1N80qSQ6tqVxe01wBrquqm1nVJU7EHrPlmfZLnMhgbvsTw1SSzByxJjfglnCQ1YgBLUiMGsCQ1YgBLUiMGsCQ18v8BukB7+Z79H08AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.catplot(x='algorithm', y='RMSE', data=user_rmse, kind='bar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "536d6fc6",
   "metadata": {
    "papermill": {
     "duration": 0.031215,
     "end_time": "2022-02-17T22:30:46.000327",
     "exception": false,
     "start_time": "2022-02-17T22:30:45.969112",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 313.546121,
   "end_time": "2022-02-17T22:30:46.451751",
   "environment_variables": {},
   "exception": null,
   "input_path": "eval-report.ipynb",
   "output_path": "eval-report.ml10m.ipynb",
   "parameters": {
    "dataset": "ml10m"
   },
   "start_time": "2022-02-17T22:25:32.905630",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}