examples/Warm_start_and_standard_scenarios.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demonstration of a simple evaluation pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Polara supports various evaluation regimes and can be tuned flexibly to achieve the setup you need.\n",
    "<div class=\"alert alert-block alert-warning\"> Note, that particular evaluation settings may not be directly supported by some models and may require a certain model modification.</div>\n",
    "\n",
    "For example, matrix factorization models are not directly applicable in the *warm-start scenario* (when test users are not present in the training set) until the *folding-in* technique is implemented for generating recommendations. Keep this in mind when creating your custom solutions.  \n",
    "\n",
    "<div class=\"alert alert-block alert-info\">In this example we demonstrate basic evaluation of two well-known factorization models - `PureSVD` and `iALS` - in both standard and warm-start settings.</div>\n",
    "\n",
    "The models can be easily tested in both scenarios without any modification due to automatic handling of the warm-start case, provided by Polara."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing Movielens-1M data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from polara.recommender.data import RecommenderData\n",
    "from polara.datasets.movielens import get_movielens_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userid</th>\n",
       "      <th>movieid</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>661</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>914</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3408</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2355</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userid  movieid  rating\n",
       "0       1     1193       5\n",
       "1       1      661       3\n",
       "2       1      914       3\n",
       "3       1     3408       4\n",
       "4       1     2355       5"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = get_movielens_data() # will automatically download it, or you can specify a path to the local copy\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000209, 3)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'permute_tops': False,\n",
       " 'warm_start': True,\n",
       " 'holdout_size': 3,\n",
       " 'test_sample': None,\n",
       " 'shuffle_data': False,\n",
       " 'random_holdout': False,\n",
       " 'negative_prediction': False,\n",
       " 'test_fold': 5,\n",
       " 'test_ratio': 0.2}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_model = RecommenderData(data, 'userid', 'movieid', 'rating', seed=0)\n",
    "data_model.get_configuration()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standard scenario with known users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing data...\n",
      "Done.\n",
      "There are 996585 events in the training and 3624 events in the holdout.\n"
     ]
    }
   ],
   "source": [
    "data_model.random_holdout = True # allow not only top-rated items in evaluation, this reduces evaluation biases\n",
    "data_model.warm_start = False # standard case\n",
    "data_model.prepare()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check for demonstration purposes that all test users are present in the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_model.test.holdout['userid'].isin(data_model.index.userid.training.new).all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PureSVD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from polara.recommender.models import SVDModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PureSVD training time: 0.149s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Relevance(precision=0.34864790286975716, recall=0.2008830022075055, fallout=0.05601545253863134, specifity=0.6227924944812362, miss_rate=0.7287527593818984),\n",
       " Ranking(nDCG=0.1426077960282924, nDCL=0.04915993850533),\n",
       " Experience(coverage=0.12169454937938479),\n",
       " Hits(true_positive=512, false_positive=112, true_negative=1164, false_negative=1836)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd = SVDModel(data_model) # create model\n",
    "svd.switch_positive = 4 # mark ratings below 4 as negative feedback and treat them accordingly in evaluation\n",
    "svd.build() # fit model\n",
    "svd.evaluate() # by default it calculates the total number of hits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### iALS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# implicit library must be installed separately, follow instructions at https://github.com/benfred/implicit \n",
    "from polara.recommender.external.implicit.ialswrapper import ImplicitALS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Intel MKL BLAS detected. Its highly recommend to set the environment variable 'export MKL_NUM_THREADS=1' to disable its internal multithreading\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iALS training time: 1.837s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Relevance(precision=0.34864790286975716, recall=0.2015728476821192, fallout=0.06084437086092715, specifity=0.6179635761589404, miss_rate=0.7280629139072847),\n",
       " Ranking(nDCG=0.14200497688853128, nDCL=0.055115784933481085),\n",
       " Experience(coverage=0.14813815434430652),\n",
       " Hits(true_positive=514, false_positive=118, true_negative=1158, false_negative=1834)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "als = ImplicitALS(data_model) # create model\n",
    "als.switch_positive = 4 # same as for PureSVD, affects only evaluation\n",
    "als.build()\n",
    "als.evaluate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The maximum possible number of correct recomendations is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2348"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_model.test.holdout.query('rating>=4').shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both models correctly retrieve around a quarter of all items. Let's look on the averaged relevance scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Relevance(precision=0.34864790286975716, recall=0.2008830022075055, fallout=0.05601545253863134, specifity=0.6227924944812362, miss_rate=0.7287527593818984)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd.evaluate('relevance')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Relevance(precision=0.34864790286975716, recall=0.2015728476821192, fallout=0.06084437086092715, specifity=0.6179635761589404, miss_rate=0.7280629139072847)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "als.evaluate('relevance')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Warm-start scenario"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will split test users from the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing data...\n",
      "19 unique movieid's within 26 testset interactions were filtered. Reason: not in the training data.\n",
      "1 unique movieid's within 1 holdout interactions were filtered. Reason: not in the training data.\n",
      "1 of 1208 userid's were filtered out from holdout. Reason: incompatible number of items.\n",
      "1 userid's were filtered out from testset. Reason: inconsistent with holdout.\n",
      "Done.\n",
      "There are 807458 events in the training and 3621 events in the holdout.\n"
     ]
    }
   ],
   "source": [
    "data_model.warm_start = True # warm-start case\n",
    "data_model.prepare()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's no intersection between test and training users:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_model.index.userid.test.old.isin(data_model.index.userid.training.old).any()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">Polara makes a certain level of efforts to preserve data sanity and consistency.</div>\n",
    "\n",
    "For example, as can be seen from the log message above, it filters out the items that are happen to be in the test split but are not a part of the training set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PureSVD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PureSVD training time: 0.107s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Relevance(precision=0.34907484120408727, recall=0.2020160176746755, fallout=0.05661419497376415, specifity=0.6219276442971554, miss_rate=0.7275614471140569),\n",
       " Ranking(nDCG=0.1425979962160517, nDCL=0.04954089721828449),\n",
       " Experience(coverage=0.12042310821806347),\n",
       " Hits(true_positive=515, false_positive=111, true_negative=1164, false_negative=1831)]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd.build()\n",
    "svd.evaluate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, that you do not have to recreate the models as they operate on top of the `data_model` instance.  \n",
    "In fact, the state change in `data_model` is synchronized with the dependent models' states. It will force models to rebuild themselves even if you do not explicitly specify it (even though it is recommended to be explicit to conform with the Zen of Python)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### iALS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Intel MKL BLAS detected. Its highly recommend to set the environment variable 'export MKL_NUM_THREADS=1' to disable its internal multithreading\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iALS model is not ready. Rebuilding.\n",
      "iALS training time: 1.615s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Relevance(precision=0.35183650925158794, recall=0.19994476663904998, fallout=0.05979011322838994, specifity=0.6187517260425297, miss_rate=0.7296326981496823),\n",
       " Ranking(nDCG=0.1389285485663426, nDCL=0.05443458388828408),\n",
       " Experience(coverage=0.14591809058855437),\n",
       " Hits(true_positive=510, false_positive=117, true_negative=1158, false_negative=1836)]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "als.evaluate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The maximum possible number of correct recomendations is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2346"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_model.test.holdout.query('rating>=4').shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check relevance scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Relevance(precision=0.34907484120408727, recall=0.2020160176746755, fallout=0.05661419497376415, specifity=0.6219276442971554, miss_rate=0.7275614471140569)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd.evaluate('relevance')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Relevance(precision=0.35183650925158794, recall=0.19994476663904998, fallout=0.05979011322838994, specifity=0.6187517260425297, miss_rate=0.7296326981496823)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "als.evaluate('relevance')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final remark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In these experiments we used the default settings for both models and, therefore, the results may not be neccessarily optimal. \n",
    "Also note, that the output of SVD is deterministic, while iALS tends to provide varying results, spread around some average value.  \n",
    "<div class=\"alert alert-block alert-warning\">In order to provide a fair comparison of these models one have to run a full cross-validation experiment with parameters tuning and confidence interval estimation.</div>\n",
    "\n",
    "It can be easily done within Polara as well and will be covered in a separate guide."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}